Reframing Taxonomy Development in Collaborative Computing Research: A Review and Synthesis of CSCW Literature 2003–2010

CSCW has been defined by interdisciplinarity from its very beginnings and its focus has changed dramatically (more in the US than in Europe), from traditional studies of work practices to more eclectic approaches such as social networks, disaster management, domestic life, and crowdsourcing [1]. With technologies pervading ordinary settings and increasingly aiding cooperative work arrangements, CSCW has a recurrent commitment on presenting different ways of thinking about technology development, evaluation, and impact on the societal framework. The importance of in-depth studies examining collaborative settings has been stressed when designing technical artifacts [2]. However, little is known about their successful contributions for technology-driven paradigmatic shifts largely geared by software and hardware industries and research fields [3]. Experimental prototypes have been developed as artifacts by developers working in the field of CSCW and pre-packaged by major vendors for widely available platforms [4]. Nevertheless, the uncertainty motivated by the pace of technology development affects the way as research communities evolve and several gaps remain unfilled concerning the examination of their complex settings.

Evaluating social and collaborative technologies becomes a critical factor to detect changes while anticipating directions as new tools are introduced and deployed [5]. Novel evaluation approaches are needed to uncover not only central aspects related with groupware attributes, success, and failure [6] but also new classes of technologies and functional attributes. Advances in CSCW have spread across journals, conference proceedings, posters, tutorials, book series, technical reports, and social networking services created for scientists. As we enter an age of increasingly larger and noisier data, the challenge here relies on the “tedious and lengthy task” of finding, analyzing and systematizing all relevant issues [9]. Hence, a research effort in the interdisciplinary field of CSCW is thus justified regarding its commitment with the development of innovative technologies by means of ethnographic studies, conceptual work, experiments, and evaluation studies.

Understanding the nature and evolution of collaboration as a social phenomenon is a difficult goal to achieve reading literature only. Nonetheless, systems developed to support or enhance collaboration activities are concrete and can be further examined. Although it seems as an extremely complex process to present an analysis comprehensive enough, a study using Mittleman et al.’s [10] taxonomy is elaborated in the form of hypothesis about the characteristics of collaborative computing in terms of application-level domains and functional attributes. The study also explores concomitant changes resulting from technical deployments to support or enhance collaborative practices. In addition, Grounded Theory [13] is also addressed as a research approach intended to extract new categories for taxonomy development.

In the remainder of this paper we first present a discussion on seminal publications, exploring a taxonomic rationale of collaborative systems. In Sect. 3, we present the methodology used to classify the functional attributes of collaborative systems from literature. Section 4 shows the main results of our study. This section also provides some exploratory remarks on the use of Grounded Theory for taxonomy development, summarizing some of the lessons learned and future directions. Section 5 summarizes and concludes this piece by providing some actionable recommendations.

Tracing the origins of CSCW, Greenberg [14] provided an annotated bibliography comprising general sources, groupware systems, and concepts. The earliest mention is to the Bush’s [15] vision of Memex as “a way of structuring and leaving trails through a large, shared information store.” Meanwhile, Augment/NLS [16] was envisioned as a research center for augmenting human capabilities through the use of artifacts, language, and training. In the literature, there are a surprising number of examples of groupware systems (e.g., Coordinator [17]). CSCW research during the 90s was focused on creating solutions flexible enough to cope with the “high flexibility, contextualization and variability of human activity” [19]. Over time, it has evolved in response to the social and technological advances, comprising foundations, approaches and languages for a better collaboration experience into multi-user environments and interfaces. As argued by Law and colleagues [20], social software tools and packages produced nowadays are similar to groupware appeared in 90s but more versatile, lightweight, wide-ranging, and dynamic. Although little is known on the adoption and real impact of collaborative systems, it is possible to assume that they had more success outside the workplace [21]. The implementation of such tools involves a “structured, continuous and flexible adaptation” affected by the tendency of users to shape the system to their specific needs [22].

Theoretical frameworks can help us characterize the field of CSCW [8, 23]. Thus, understanding major technology shifts becomes a critical endeavor in a highly volatile community predominantly established on time frames of evolutionary stasis disrupted by short periods of sudden technological waves [3]. Only a small number of studies have already examined impacts on work practices when a collaborative system is introduced. As outlined by Pinelle and Gutwin [4], “a better understanding of how groupware systems have been evaluated in the past can help to frame the discussion of what methods and techniques should be considered for future evaluations”. The authors preceded the advent of the 21st century with a review of papers that introduced or evaluated a groupware system from ACM CSCW conference proceedings. Findings included a predominance of synchronous applications and a small number of deployments in real world settings when comparing with academic or research implementations. Furthermore, an emphasis on ongoing evaluations of prototypes comparing to evaluations of completed software packages was also denoted.

A vast set of classification frameworks were proposed in order to organize contributions, “but none of them are comprehensive enough since they focus either on a particular aspect of collaboration or on the specific mechanism that the tools follow” [24]. Such frameworks have several elements in common, including group and individual characteristics, task properties, situation factors, and group processes and outcomes [6]. In the light of this work, Bafoutsou and Mentzas [25] emphasized a number of areas of further consideration that arise when studying collaborative applications, including classification dimensions such as scalability and usability. Evolutionary approaches based on requirements engineering were also proposed for scaffolding groupware implementation [26]. Sarma and colleagues [24] adapted Maslow’s hierarchy of needs pyramid to create a classification framework for characterizing collaboration needs. A multifaceted evaluation framework covering relationships underlying communication, collaboration, coordination, work coupling, and joint awareness was also provided [6]. A somewhat similar body of work has sought to propose typologies of collaborative tool capabilities [28], while Antunes et al. [8] proposed a framework for collaborative systems evaluation covering all stages of system development. Further research was reviewed by Grudin and Poltrock [29] regarding the development of taxonomic schemes comprising technology features and collaborative behavior.

As we enter an age of increasingly larger and noisier data, too many relevant results and work are hidden in the large volumes of literature annually published. Data serve as evidence to support scientific inquiry and researchers guide their pursuits around data as a foundation for scientific collaboration [30]. CSCW applications are difficult to evaluate [6] since most of the taxonomies existing at the time are inadequate to classify more complex systems that include a large variety of features. As collaborative systems and technologies evolve and become more complex, it is much harder to evaluate them appropriately with a clear perception of their practical implications.

This work relies on the premise that despite technologies may change frequently, classes of systems may endure for much longer [31]. It is also assumed that previous results and implications for designing a system can also be useful on the design of another type of system. Classify papers according to a taxonomy is one of the stages of the formal process outlined by Kitchenham [32] for conducting a Systematic Literature Review (SLR). We followed evidence-based research methods for conducting a feature analysis and collaborative systems evaluation [5, 34]. Thus, the strengths and weaknesses of collaborative computing technology, in terms of how well they provided each of the functional attributes, are discussed. For the purpose of this study, we also discuss the use of Grounded Theory [13] as a “railroad” for CSCW research [35].

3.2 Classification Process

The taxonomy used for analysis was adopted from Mittleman and colleagues [10] to shortly compare collaborative systems and research prototypes together with the functionalities implemented within each system. The selection of this taxonomy was based on a systematic review of evaluation frameworks [36]. The classification scheme for collaborative technologies [10] divides a set of application-level domains into jointy authored pages, streaming tools, information access tools, and aggregated systems. The last category consists of a combination of the first three types of technologies in order to optimize them while supporting cooperative work practices in settings for which is necessary more than a single technology. On the other hand, the comparison scheme relies on nine core capabilities for collaboration affordances. First, the core functionality provided by a tool can range from creating and/or sharing a single text page to video/audio stream. Content describes the type of contributions or data structures. Examples include text messages, URL, pictures, data stream, and hypermedia. Relationships among contributions can range from collection to list, tree, and graph. Moreover, supported actions represent the things that users can do on content and relations (e.g., modify content, remove data, receive contributions). Synchronicity can be explained by when participants are working at the same time (synchronous) or different time (asynchronous). Identifiability is another action parameter characterized by the degree to which contributors can determine who executed an action (e.g., full anonymity). Access control deals with the configuration of user’s privileges to execute actions, while session persistence is the degree to which contributions persist or disappear in the system. Alert mechanisms deal with the interruptions (notifications) suffered by the user when a new contribution is made into the system. Finally, awareness is the perception of users about what each member develops and the contextual knowledge that they have about what is happening within the system [10].

Feature analysis is a recognized evaluation method in software engineering and can be understood as “a qualitative form of evaluation involving the subjective assessment of the relative importance of different features plus an assessment of how well each of the features is implemented by the candidate tools” [34]. The classification process consisted on gathering descriptive metadata related to a publication and adding contextual knowledge to each record. As can be seen in Fig. 1, a total of 1480 papers published in CSCW devoted venues between 2003 and 2010 were indexed by year, ID, name of the author(s), title of the paper, and conference categorization using DBLP and ACM Digital Library information sources. The classification and comparison schemes for collaborative systems [10] were thus applied on 541 papers speculating on the social and organizational impact of collaboration technologies. Such primary studies include the design, deployment, and evaluation of new (or already introduced) systems, tools, and architectures. Once data were categorized and organized, each paper was screened and evaluated taking into account the classes and functional attributes that were either present or absent [34]. The sample was then revisited using Grounded Theory [13] as an experimental approach for extracting new categories.

Open image in new window

As a research community mainly constituted by behavioral scientists and system developers, CSCW has a long tradition in conceptualizing collaboration dynamics and proposing technology-based tools. The development of communication networks led to the emergence of social interaction support systems in a broad-spectrum of application domains ranging from healthcare and emergency response to ludic scenarios [21]. The analysis of the functional attributes by conference (Fig. 2) demonstrates an interest of ECSCW by awareness, notification mechanisms, and access controls. Our results go beyond previous reports, showing that the ACM CSCW conference discarded awareness in some systems but makes use of alert mechanisms and permanent contributions. A similar pattern of results was obtained in CRIWG proceedings from 2006–2008. In turn, ACM GROUP shows greater emphasis on text sharing with hyperlinks and asynchronous tools, representing a major outlet to study awareness. Nevertheless, CSCW is a very dynamic field and the research focus continually changes over time. As the analyzed periods are different for each conference, some differences can be due to the time periods rather than to conceptual focuses.

Open image in new window

Work processes in cooperative ensembles require a coordination of resources with high levels of interdependency between tasks [40]. As a socially oriented process, cooperative work is ordinarily enabled by a shared database, requiring an active construction process by participants into a mutual information working space to reach a common ground [41]. Text sharing was the most pronounced feature in our sample. For instance, electronic patient record systems, three-dimensional digital media design environments, and activity-tracking tools promote social interaction and engagement through text sharing. In addition, collaborative applications with both text sharing and conferencing features demonstrated growing indicators at the end of the previous decade. Examples include avatar-based meeting support tools and robots with computer vision such as Lunar rover robot [39]. The results found clear support for hypermedia as the most adopted type of data structure that may be used to a particular collaboration. On the other hand, we speculate that the decrease of data stream might be due to the expansion of this type of functionality provided by WWW. When extrapolating to the different associations that users can establish among contributions, collection was the most visible implementation by programmers of collaborative systems. This indicator can reinforce the notion of a lack of structure in certain components of collaboration technology. Adding content (e.g., a new blog entry) is noticeably high, being present in tools like ActivitySpotter [42]. Results also showed a high expression in the ability to comment in groupware systems. For instance, a group of users can produce annotations in mutual digital documents and support decision-making while reducing the cost of reading a document [27].

The present study confirmed the Pinelle and Gutwin’s [4] findings about the prevalence of synchronous systems in the collaborative work sphere. Such kind of systems were followed by tools with both synchronous and asynchronous features. In addition, asynchronous applications were then slightly introduced, demonstrating a growth and optimistic perspectives reflected in platforms such as Amazon Mechanical Turk¹. This finding is in line with the fact that the conception of collaboration has changed during the last decade, being more multi-tasking, asynchronous and flexible given the opportunities that recent technology offers to the users [23]. Although collaborative systems have a strong focus on identifying users who perform shared actions, tools where the user acts anonymously or can choose to be anonymous or identified have shown indicators of remarkable growth. Access controls denoted preeminence signs on the inclusion of authentication mechanisms. With this feature, security and privacy issues can be enhanced by preventing unauthorized or malicious access to resources and users may increase their trust using collaboration technology. From the data in Fig. 2, it is apparent that there was little difference between tools providing awareness compared to those that did not make use of this feature. However, awareness has been a critical topic in CSCW research and developers should be mainly focused on the implementation of this feature appropriately. From this data, we can also see high values for notification/alert mechanisms and permanent sessions.

Collaborative computing research has been expanded from studying group work in organizations or workplaces to the home, including the effects of life disruptions on home technology routines [37]. Social networking has been also a topic of intensive interest at work [38]. Technologies where users can edit text collaboratively and use hyperlinks have been growing in the literature. Some examples are online social tagging, gift exchange and mind mapping/brainstorming systems (e.g., GroupMind [18]). A preference by group dynamics and a steady increase of social tagging, video conferencing, and search engines were noted. Moreover, audio conferencing and desktop/application sharing tools clearly express a decrease. Aggregated systems showed a peak in 2005 before decreasing in the last years, while shared file repositories grown up in 2009. Syndication tools followed the same path, and conversation systems were more addressed between 2003 and 2004. Nevertheless, polling tools had more influence in 2007, while shared editors remained stable all years. Figure 3 details the results of our analysis on the classes of collaborative systems and technologies by venue.

Open image in new window

Concerning the scope of ECSCW, this venue addressed desktop/application sharing, followed by aggregated systems and conversation tools with remarkable indicators on video conferencing. Nevertheless, ACM CSCW put the focus on polling tools, conversation tools, shared editors, desktop/application sharing, and syndication tools. On average, we found many studies spread over conversation systems, social tagging tools, aggregated systems, search engines, and shared editors in ACM GROUP conference proceedings. Interestingly, CRIWG showed a large number of group dynamics tools, aggregated systems, polling tools, and shared editors. Curiously, all research venues published most of their studies on group dynamics. Regarding the categories with the lowest values (e.g., audio conferencing), this may be related with a focus of intensive research on these systems until the beginning of the 21st century, resulting in a consequent slowdown by achieving stabilized solutions. Oppositely, an explosion in the use of social networking sites, question answering systems, social bookmarking, microblogs, wikis, virtual worlds, crowd computing, and Web conferencing tools brought opportunities for studying online behavior. This fact is also consistent with the growth of social tagging systems, search engines, video conferencing, and syndication applications.

4.1 Grounded Theory for Taxonomy Development in CSCW: A ‘Sleeping Dragon’ Wanting for Awakening

A taxonomic evaluation can characterize the most addressed systems and functional attributes by venue. However, further research is needed updating taxonomic categories in a systematic way considering the changing requirements of collaboration. From a general examination on DBLP, both ACM CSCW and ACM GROUP divide their tracks into particular categories that range from collaborative software development to crowdsourcing and citizen science, healthcare and emergency response, social media, tangible interfaces, home and family technology, Wikipedia studies, MOOCs, games and virtual worlds, policy, cybersecurity, Q&A sites, etc. Conversely, ECSCW publications are mostly based on field-based studies using methods intended to inform the development of technologies from the ground up, presenting fieldwork on multiple scenarios such as heritage and homecare.

Despite their utility, such categories are very generic, being difficult to obtain qualitative data with high level of detail. Measuring scientific production is a hard task that needs a comprehensive, theoretically grounded and practically valuable conceptualization of its structure and evolution [33]. Despite the efforts in the creation of conceptual frameworks, taxonomies, ontologies, and thesauri to represent knowledge about consolidated and emerging topics, existing classification systems fail on capturing the intellectual structure of a scientific field, understood as an abstraction of the collective knowledge of its researchers. The main challenge relies on the following question: How do researchers, venture capital investors, program managers and students keep up with advances in the expanding field of CSCW? This is particularly complex to manage since we are nowadays overwhelmed by findings of thousands of scholars. Long-term classification of massive data collections can be considered as a challenging issue for research communities, institutions and disciplines regarding the inherent difficulty of recognizing gaps, trends, patterns, concepts, “ghost theories”, and social-technical aspects.

We assume that Grounded Theory [13], a methodology originated in sociology, can be particularly useful in CSCW (as discussed by Muller and Kogan [35]) to make sense of large volumes of data from scientific publications. In the particular structure of work in which an individual is making sense of literature, both domain and type of data to be extracted are previously known. However, new categories can emerge from distinct ways we analyze our data, as well the strategies we use for finding evidence. Based on the Muller and Kogan’s [35] guidelines for working with data in Grounded Theory, we returned to our initial sample and categorized three years of research in collaborative computing through a series of data-driven operations conducted to develop a high-quality description in which open and axial codes were gradually developed and grouped into categories and dimensions of analysis. Table 2 provides an example of the descriptive codes used to perform this exploratory analysis. Each paper was screened and a set of categories was then extracted and organized on the basis of a semantic relationship to the data presented in the collection form.

Table 2.

Data collection form (adapted from Muller and Kogan [35]).

Paper ID	P1
Source	ACM GROUP
Year	2010
General concepts and terms	Specialized division of labor Modularized division of labor Essential complexity Online creative collaboration Peer production Online communities
Contributions	Systematic research agenda Success factors in open-source software projects
Results, insights, and implications for design	Large software projects require many skill sets and, therefore, many people Success factors in open-source software projects include planning and structure, reputation and experience, communication and dedication Success rates were low for both open source software projects and collabs (less than 20%)
Method	Qualitative (semi-structured interviews and participant observation)
Sample	N = 17 animators (amateurs, students, and professionals), age (16 to 29), genre (male) N = 892 collab threads
Variables	Ratings and popularity of completed collabs Themes as content guidelines for the project (e.g., music video, single narrative, comedy sketches) Specs as technical specifications for the project (e.g., dimensions, frame rate, audio streaming) Authorship criteria determining which artists get “co-author” credit (e.g., best submissions, most helpful, voting) Restrictions identifying prohibited content types for the project (e.g., violence, nudity, profanity) Gatekeeping rules govern who can join the project (e.g., leader’s pick, experience baselines, tryouts) Communication preferences providing instructions for contacting the leader (e.g., email, IM, phone/voice)
Research agenda	What are the underlying principles allowing online creative collaboration to succeed, and how well do they transfer from one domain to another? Although the applicability of identified principles to open-source software and collabs is promising, more work is needed to test them in still other domain
System(s) proposed or evaluated	Newgrounds
Application-level domain	Multi-author system for animated movies and games
Features and functional attributes	Post a collab thread describing the project and inviting community members to join Submit completed animations for judgment by other members
Other tools mentioned	Wikipedia SourceForge Slashdot
Notes	Roles (leader, artist, co-author) Interview quote#1 (“A collaboration can’t succeed without [communication]”)

After performing the extraction of categories from a total of 145 publications from ACM GROUP 2010 (36), ECSCW 2009 (23), and ACM CSCW 2008 (86), we formulate a set of considerations. For taxonomy development, it was possible to obtain new categories that can be added to current classification schemes (e.g., [10]) while contributing to the development of a rigorous and comprehensive evaluation framework for collaborative computing. For instance, selective undo in collaborative applications appeared as a new functional attribute. Specialized and modularized division of labor was associated to success factors in online creative collaboration (peer production) in the context of collaborative animated movies and open source software development. Perhaps, concepts like articulation work and coordination by avoidance can be ignored from new researchers in the field. Thus, we consider the record and contextualization extremely important. We also found possible associations between terms like genre ecologies ↔ community regulation, participatory citizenship ↔ timeline collaboration, virtual worlds ↔ phatic communication, version control ↔ collaborative conflict resolution, boundary negotiating artifacts ↔ cyberinfrastructure, and workspace awareness ↔ real-time collaborative 3D design. Table 3 summarizes the main categories extracted from literature in this preliminary approach to the use of Grounded Theory [13] for CSCW research evaluation.

Table 3.

General concepts, methods, and system attributes identified from CSCW literature.

Open image in new window

Open image in new window

New types of systems emerged from applying Grounded Theory [13]. Some examples are games, home technology, or tabletop displays. Describe a system is also relevant for who are interested on experimenting a certain tool for particular purposes considering its main features described with high detail. From scheduling patients in a medical information system to collaborative filtering, several features have already appeared. In general, Grounded Theory [13] can be particularly useful as a method for unveiling new kinds of taxonomic units from publication records with higher levels of granularity. However, researchers in HCI and CSCW will need to inform their choices and problems to be solved with a strong knowledge of the research literature [35].

Understanding CSCW as a socio-technical design space is a big challenge. While making sense of technology can be a relatively easy task, collaboration represents a phenomenon that requires intensive analysis of concepts and contexts. This study explored collaborative systems evaluation techniques by analyzing a wide variety of publications from ACM CSCW, ACM GROUP, ECSCW, and CRIWG. However, this paper does not represent an attempt to summarize the state-of-the-art in collaborative computing research. On the other hand, it provides an initial springboard for the analysis of large volumes of scientific data published every year through a dynamic interplay between a set of strategies and methods. In the future, we aim to evaluate collaborative systems and cooperative work conceptualizations from multivariate views. In this context, analytical techniques and methods must be reinvented systematically and recursively to cope with the advancements in CSCW due to its interdisciplinary nature. Therefore, it is critical to understand how data itself is subjectively situated and used in other disciplines [30]. This breadth raises a number of questions, including the challenge of understanding the structure and evolution of science by reviewing the past while developing current conceptualizations and models of scholarly activity [33]. Finally, we can also assume that distilling such data using Grounded Theory [13] while adopting a mixed-initiative approach based on machine learning and crowdsourcing can constitute an effective approach to cope with the different social-technical dimensions in the field of CSCW while helping researchers and the general public to be aware of its evolution.