Advertisement

Crowd Research: Open and Scalable University Laboratories

  • Rajan VaishEmail author
  • Snehalkumar (Neil) S. Gaikwad
  • Geza Kovacs
  • Andreas Veit
  • Ranjay Krishna
  • Imanol Arrieta Ibarra
  • Camelia Simoiu
  • Michael Wilber
  • Serge Belongie
  • Sharad C. Goel
  • James Davis
  • Michael S. Bernstein
Chapter
Part of the Understanding Innovation book series (UNDINNO)

Abstract

Research experiences today are limited to a privileged few at select universities. Providing open access to research experiences would enable global upward mobility and increased diversity in the scientific workforce. How can we coordinate a crowd of diverse volunteers on open-ended research? How could a PI have enough visibility into each person’s contributions to recommend them for further study? We present Crowd Research, a crowdsourcing technique that coordinates open-ended research through an iterative cycle of open contribution, synchronous collaboration, and peer assessment. To aid upward mobility and recognize contributions in publications, we introduce a decentralized credit system: participants allocate credits to each other, which a graph centrality algorithm translates into a collectively-created author order. Over 1500 people from 62 countries have participated, 74% from institutions with low access to research. Over 2 years and three projects, this crowd has produced articles at top-tier Computer Science venues, and participants have gone on to leading graduate programs.

Keywords

Crowdsourcing Citizen science Crowd research 

Notes

Acknowledgements

We thank over 1500 members of the Stanford Crowd Research Collective community for their contributions. This work was supported by Office of Naval Research awards N00014-16-1-2894 and N00014-15-1-2711, Institute for Scalable Scientific Data Management at UCSC and Los Alamos National Laboratory, Toyota, and the Hasso-Plattner Design Thinking Research Program.

References

  1. Baeza-Yates, R. A., Castillo, C., López, V., Shubik, M., Hopcroft, J., & Sheldon, D. (2007). Pagerank increase under different collusion topologies. In AIRWeb (Vol. 5, pp. 68–81). New York: Springer.Google Scholar
  2. Bianchini, J. A. (2011). Expanding underrepresented minority participation: America’s science and technology talent at the crossroads. Washington: National Academies.Google Scholar
  3. Bonney, R., Ballard, H., Jordan, R., McCallie, E., Phillips, T., Shirk, J., & Wilderman, C. C. (2009). Public participation in scientific research: Defining the field and assessing its potential for informal science education.Google Scholar
  4. Bowen, W. G., & Bok, D. (2016). The shape of the river: Long-term consequences of considering race in college and university admissions. Princeton: Princeton University Press.Google Scholar
  5. Budhathoki, N. (2016). Who are the mappers and why do they map in OpenStreetMap. https://www.youtube.com/watch?v=LvakiUOsDrM
  6. Chen, Y., Oney, S., & Lasecki, W. S. (2016). Towards providing on-demand expert support for software developers. In Proceedings of the 2016 CHI conference on human factors in computing systems (pp. 3192–3203). New York: ACM.CrossRefGoogle Scholar
  7. Cooper, S., Khatib, F., Treuille, A., Barbero, J., Lee, J., Beenen, M., Leaver-Fay, A., Baker, D., Popovic, Z., & Players, F. (2010). Predicting protein structures with a multiplayer online game. Nature, 466(7307), 756–760.CrossRefGoogle Scholar
  8. Cox, J., Eun Young, O., Simmons, B., Lintott, C., Masters, K., Greenhill, A., Graham, G., & Holmes, K. (2015). Defining and measuring success in online citizen science: A case study of Zooniverse projects. Computing in Science & Engineering, 17(4), 28–41.CrossRefGoogle Scholar
  9. Cramton, C. D., & Hinds, P. J. (2004). Subgroup dynamics in internationally distributed teams: Ethnocentrism or cross-national learning? Research in Organizational Behavior, 26, 231–263.CrossRefGoogle Scholar
  10. Cranshaw, J., & Kittur, A. (2011). The polymath project: Lessons from a successful online collaboration in mathematics. In Proceedings of the SIGCHI conference on human factors in computing systems (pp. 1865–1874). New York: ACM.Google Scholar
  11. Cronje, R., Rohlinger, S., Crall, A., & Newman, G. (2011). Does participation in citizen science improve scientific literacy? A study to compare assessment methods. Applied Environmental Education & Communication, 10(3), 135–145.CrossRefGoogle Scholar
  12. Dillahunt, T. R., Chen, B., & Teasley, S. (2014). Model thinking: Demographics and performance of MOOC students unable to afford a formal education. In Proceedings of the first ACM conference on learning@ scale conference (pp. 145–146). New York: ACM.CrossRefGoogle Scholar
  13. Dillahunt, T. R., Ng, S., Fiesta, M., & Wang, Z. (2016). Do massive open online course platforms support employability? In Proceedings of the 19th ACM conference on computer-supported cooperative work & social computing (pp. 232–243). New York: ACM.Google Scholar
  14. Ding, Y., Yan, E., Frazho, A., & Caverlee, J. (2009). PageRank for ranking authors in co-citation networks. Journal of the American Society for Information Science and Technology, 60(11), 2229–2243.CrossRefGoogle Scholar
  15. Dow, S., Kulkarni, A., Klemmer, S., & Hartmann, B. (2012). Shepherding the crowd yields better work. In Proceedings of the ACM 2012 conference on computer supported cooperative work (p. 1013). New York: ACM.Google Scholar
  16. Eric Landrum, R., & Nelsen, L. R. (2002). The undergraduate research assistantship: An analysis of the benefits. Teaching of Psychology, 29(1), 15–19.CrossRefGoogle Scholar
  17. Gaikwad, S. S., Morina, D., Ginzberg, A., Mullings, C., Goyal, S., Gamage, D., Diemert, C., Burton, M., Zhou, S., Whiting, M., Ziulkoski, K., Ballav, A., Gilbee, A., Niranga, S. S., Sehgal, V., Lin, J., Kristianto, L., Regino, J., Chhibber, N., Majeti, D., Sharma, S., Mananova, K., Dhakal, D., Dai, W., Purynova, V., Sandeep, S., Chandrakanthan, V., Sarma, T., Matin, S., Nassar, A., Nistala, R., Stolzoff, A., Milland, K., Mathur, V., Vaish, R., & Bernstein, M. S. (2016). Boomerang: Rebounding the consequences of reputation feedback on crowdsourcing platforms. In Proceedings of the 29th annual ACM symposium on user interface software and technology (UIST ’16). New York: ACM.Google Scholar
  18. Gaikwad, S. S., Morina, D., Nistala, R., Agarwal, M., Cossette, A., Bhanu, R., Savage, S., Narwal, V., Rajpal, K., Regino, J., Mithal, A., Ginzberg, A., Nath, A., Ziulkoski, K. R., Cossette, T., Gamage, D., Richmond-Fuller, A., Suzuki, R., Herrejón, J., Le, K., Flores-Saviaga, C., Thilakarathne, H., Gupta, K., Dai, W., Sastry, A., Goyal, S., Rajapakshe, T., Abolhassani, N., Xie, A., Reyes, A., Ingle, S., Jaramillo, V., Godinez, M., Angel, W., Toxtli, C., Flores, J., Gupta, A., Sethia, V., Padilla, D., Milland, K., Setyadi, K., Wajirasena, N., Batagoda, M., Cruz, R., Damon, J., Nekkanti, D., Sarma, T., Saleh, M., Gongora-Svartzman, G., Bateni, S., Barrera, G. T., Pena, A., Compton, R., Aariff, D., Palacios, L., Ritter, M. P., Nisha, K. K., Kay, A., Uhrmeister, J., Nistala, S., Esfahani, M., Bakiu, E., Diemert, C., Matsumoto, L., Singh, M., Patel, K., Krishna, R., Kovacs, G., Vaish, R., & Bernstein, M. (2015). Daemo: A self-governed crowdsourcing marketplace. In Adjunct proceedings of the 28th annual ACM symposium on user interface software & technology (pp. 101–102). New York: ACM.CrossRefGoogle Scholar
  19. Gaikwad, S. S., Whiting, M. E., Gamage, D., Mullings, C. A., Majeti, D., Goyal, S., Gilbee, A., Chhibber, N., Ginzberg, A., Richmond-Fuller, A., Matin, S., Sehgal, V., Sarma, T. S., Nasser, A., Ballav, A., Regino, J., Zhou, S., Mananova, K., Srinivas, P., Ziulkoski, K., Dhakal, D., Stolzoff, A., Niranga, S. S., Salih, M. H., Sinha, A., Vaish, R., & Bernstein, M. S. (2017). The daemo crowdsourcing marketplace. In Companion of the 2017 ACM conference on computer supported cooperative work and social computing (pp. 1–4). New York: ACM.Google Scholar
  20. Gerstner, C. R., & Day, D. V. (1997). Meta-analytic review of leader-member exchange theory: Correlates and construct issues. Journal of Applied Psychology, 82(6), 827–844.CrossRefGoogle Scholar
  21. Gilbert, E. (2013). Widespread underprovision on Reddit. In Proceedings of the 2013 conference on computer supported cooperative work (pp. 803–808). New York: ACM.Google Scholar
  22. Gowers, T., & Nielsen, M. (2009). Massively collaborative mathematics. Nature, 461(7266), 879–881.CrossRefGoogle Scholar
  23. Gurin, P., Dey, E., Hurtado, S., & Gurin, G. (2002). Diversity and higher education: Theory and impact on educational outcomes. Harvard Educational Review, 72(3), 330–367.CrossRefGoogle Scholar
  24. Haizi, Y., Deka, B., Talton, J. O., & Kumar, R. (2016). Accounting for taste: Ranking curators and content in social networks. In Proceedings of the 2016 CHI conference on human factors in computing systems (pp. 2383–2389). New York: ACM.Google Scholar
  25. Hathaway, R. S., Nagda, B. A., & Gregerman, S. R. (2002). The relationship of undergraduate research participation to graduate and professional education pursuit: An empirical study. Journal of College Student Development, 43(5), 614.Google Scholar
  26. Hinds, P. J., & Bailey, D. E. (2003). Out of sight, out of sync: Understanding conflict in distributed teams. Organization Science, 14(6), 615–632.CrossRefGoogle Scholar
  27. Hinds, P. J., & Mortensen, M. (2005). Understanding conflict in geographically distributed teams: The moderating effects of shared identity, shared context, and spontaneous communication. Organization Science, 16(3), 290–307.CrossRefGoogle Scholar
  28. Hong, L., & Page, S. E. (2004). Groups of diverse problem solvers can outperform groups of high-ability problem solvers. Proceedings of the National Academy of Sciences, 101(46), 16385–16389.CrossRefGoogle Scholar
  29. Hunter, A.-B., Laursen, S. L., & Seymour, E. (2007). Becoming a scientist: The role of undergraduate research in students’ cognitive, personal, and professional development. Science Education, 91(1), 36–74.CrossRefGoogle Scholar
  30. Irani, L. C., & Silberman, M. (2013). Turkopticon: Interrupting worker invisibility in amazon mechanical turk. In Proceedings of the SIGCHI conference on human factors in computing systems (pp. 611–620). New York: ACM.CrossRefGoogle Scholar
  31. Jordan Raddick, M., Bracey, G., Gay, P. L., Lintott, C. J., Murray, P., Schawinski, K., Szalay, A. S., & Vandenberg, J. (2009). Galaxy zoo: Exploring the motivations of citizen science volunteers. arXiv preprint arXiv:0909.2925.Google Scholar
  32. Kevin Eagan, M., Hurtado, S., Chang, M. J., Garcia, G. A., Herrera, F. A., Garibay, J. C., & Garibay, J. C. (2013). Making a difference in science education: The impact of undergraduate research programs. American Educational Research Journal, 50(4), 683–713.CrossRefGoogle Scholar
  33. Khatib, F., DiMaio, F., Cooper, S., Kazmierczyk, M., Gilski, M., Krzywda, S., Zabranska, H., Pichova, I., Thompson, J., Popovic, Z., Jaskolski, M., & Baker, D. (2011). Crystal structure of a monomeric retroviral protease solved by protein folding game players. Nature Structural & Molecular Biology, 18(10), 1175–1177.CrossRefGoogle Scholar
  34. Kiesler, S., Kraut, R., Resnick, P., & Kittur, A. (2012). Regulating behavior in online communities. In Building successful online communities: Evidence-based social design. Cambridge: MIT.Google Scholar
  35. Kittur, A., Nickerson, J. V., Bernstein, M., Gerber, E., Shaw, A., Zimmerman, J., Lease, M., & Horton, J. (2013). The future of crowd work. In Proceedings of the 2013 conference on computer supported cooperative work (p. 1301). New York: ACM.Google Scholar
  36. Kittur, A., Smus, B., Khamkar, S., & Kraut, R. E. (2011). Crowdforge: Crowdsourcing complex work. In Proceedings of the 24th annual ACM symposium on user interface software and technology (pp. 43–52). New York: ACM.Google Scholar
  37. Kizilcec, R. F., & Halawa, S. (2015). Attrition and achievement gaps in online learning. In Proceedings of the second ACM conference on learning@ scale (pp. 57–66). New York: ACM.Google Scholar
  38. Kleinberg, J. M. (1999). Authoritative sources in a hyperlinked environment. Journal of the ACM (JACM), 46(5), 604–632.CrossRefGoogle Scholar
  39. Kulkarni, C. E., Bernstein, M. S., & Klemmer, S. R. (2015). PeerStudio: Rapid peer feedback emphasizes revision and improves performance. In Proceedings of the second (2015) ACM conference on learning@ scale (pp. 75–84). New York: ACM.Google Scholar
  40. Kulkarni, C., Wei, K. P., Le, H., Chia, D., Papadopoulos, K., Cheng, J., Koller, D., & Klemmer, S. R. (2013). Peer and self assessment in massive online classes. ACM Transactions on Computer-Human Interaction (TOCHI), 20(6), 33.CrossRefGoogle Scholar
  41. Lakhani, K. R., Lifshitz-Assaf, H., & Tushman, M. (2013). Open innovation and organizational boundaries: Task decomposition, knowledge distribution and the locus of innovation. In Handbook of economic organization: Integrating economic and organizational theory (pp. 355–382). Cheltenham: Edward Elgar.Google Scholar
  42. Land, K., Slosar, A., Lintott, C., Andreescu, D., Bamford, S., Murray, P., Robert, N., Jordan Raddick, M., Schawinski, K., Szalay, A., Thomas, D., & Vandenberg, J. (2008). Galaxy zoo: The large-scale spin statistics of spiral galaxies in the sloan digital sky survey. Monthly Notices of the Royal Astronomical Society, 388(4), 1686–1692.CrossRefGoogle Scholar
  43. Lashinsky, A. (2012). Inside apple: How America’s most admired–and secretive–company really works. London: John Murray.Google Scholar
  44. LaToza, T. D., Ben Towne, W., Adriano, C. M., & Van Der Hoek, A. (2014). Microtask programming: Building software with a crowd. In Proceedings of the 27th annual ACM symposium on user interface software and technology (pp. 43–54). New York: ACM.Google Scholar
  45. Law, E., Wiggins, A., Gray, M. L., & Williams, A. (2017). Crowdsourcing as a tool for research: Implications of uncertainty. In Proceedings of the 20th ACM conference on computer-supported cooperative work & social computing (CSCW ’17). New York: ACM.Google Scholar
  46. Lee, J., Kladwang, W., Lee, M., Cantu, D., Azizyan, M., Kim, H., Limpaecher, A., Gaikwad, S., Yoon, S., Treuille, A., et al. (2014). RNA design rules from a massive open laboratory. Proceedings of the National Academy of Sciences, 111(6), 2122–2127.CrossRefGoogle Scholar
  47. Li, J. (2003). US and Chinese cultural beliefs about learning. Journal of Educational Psychology, 95(2), 258.CrossRefGoogle Scholar
  48. Lixiu, Y., Kittur, A., & Kraut, R. E. (2014). Distributed analogical idea generation: Inventing with crowds. In Proceedings of the 32nd annual ACM conference on human factors in computing systems (pp. 1245–1254). New York: ACM.Google Scholar
  49. Lopatto, D. (2004). Survey of undergraduate research experiences (SURE): First findings. Cell Biology Education, 3(4), 270–277.CrossRefGoogle Scholar
  50. Mao, A., Mason, W., Suri, S., Watts, D. J., & Malone, T. W. (2016). An experimental study of team size and performance on a complex task. PLoS One, 11(4), e0153048.CrossRefGoogle Scholar
  51. Martin, D., Hanrahan, B. V., O’Neill, J., & Gupta, N. (2014). Being a turker. In Proceedings of the 17th ACM conference on computer supported cooperative work & social computing (pp. 224–235). New York: ACM.Google Scholar
  52. Miller, R. C., Zhang, H., Gilbert, E., & Gerber, E. (2014). Pair research: Matching people for collaboration, learning, and productivity. In Proceedings of the 17th ACM conference on computer supported cooperative work & social computing (pp. 1043–1048). New York: ACM.Google Scholar
  53. Mysore, A. S., Yaligar, V. S., Ibarra, I. A., Simoiu, C., Goel, S., Arvind, R., Sumanth, C., Srikantan, A., Bhargav, H. S., Pahadia, M., Dobha, T., Ahmed, A., Shankar, M., Agarwal, H., Agarwal, R., Anirudh-Kondaveeti, S., Arun-Gokhale, S., Attri, A., Chandra, A., Chilukur, Y., Dharmaji, S., Garg, D., Gupta, N., Gupta, P., Jacob, G. M., Jain, S., Joshi, S., Khajuria, T., Khillan, S., Konam, S., Kumar-Kolla, P., Loomba, S., Madan, R., Maharaja, A., Mathur, V., Munshi, B., Nawazish, M., Neehar-Kurukunda, V., Nirmal-Gavarraju, V., Parashar, S., Parikh, H., Paritala, A., Patil, A., Phatak, R., Pradhan, M., Ravichander, A., Sangeeth, K., Sankaranarayanan, S., Sehgal, V., Sheshan, A., Shibiraj, S., Singh, A., Singh, A., Sinha, P., Soni, P., Thomas, B., Varma-Dattada, K., Venkataraman, S., Verma, P., & Yelurwar, I. (2015). Investigating the “wisdom of crowds” at scale. In Adjunct proceedings of the 28th annual ACM symposium on user interface software & technology (UIST ’15 adjunct) (pp. 75–76). New York: ACM.CrossRefGoogle Scholar
  54. Nebeling, M., To, A., Guo, A., de Freitas, A. A., Teevan, J., Dow, S. P., & Bigham, J. P. (2016). WearWrite: Crowd-assisted writing from smartwatches. In Proceedings of the 2016 CHI conference on human factors in computing systems (pp. 3834–3846). New York: ACM.CrossRefGoogle Scholar
  55. Oliveira, N., Jun, E., & Reinecke, K. (2017). Citizen science opportunities in volunteer-based online experiments. In Proceedings of the 2017 CHI conference on human factors in computing systems (pp. 6800–6812). New York: ACM.CrossRefGoogle Scholar
  56. Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349, 6251.CrossRefGoogle Scholar
  57. Page, S. E. (2008). The difference: How the power of diversity creates better groups, firms, schools, and societies. Princeton: Princeton University Press.Google Scholar
  58. Page, L., Brin, S., Motwani, R., & Winograd, T. (1999). The PageRank citation ranking: Bringing order to the web. Stanford: Stanford InfoLab.Google Scholar
  59. Pandey, V., Amir, A., Debelius, J., Hyde, E. R., Kosciolek, T., Knight, R., & Klemmer, S. (2017). Gut instinct: Creating scientific theories with online learners. In Proceedings of the 2017 CHI conference on human factors in computing systems (pp. 6825–6836). New York: ACM.CrossRefGoogle Scholar
  60. Patterson, D. A. (1994). How to have a bad career in research/academia. In Keynote, 1994 USENIX symposium on operating system design and implementation.Google Scholar
  61. Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016). Squad: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 conference on empirical methods in natural language processing.Google Scholar
  62. Reinecke, K., & Gajos, K. Z. (2015). LabintheWild: Conducting large-scale online experiments with uncompensated samples. In Proceedings of the 18th ACM conference on computer supported cooperative work & social computing (pp. 1364–1378). New York: ACM.Google Scholar
  63. Retelny, D., Robaszkiewicz, S., To, A., Lasecki, W. S., Patel, J., Rahmati, N., Doshi, T., Valentine, M., & Bernstein, M. S. (2014). Expert crowdsourcing with flash teams. In Proceedings of the 27th annual ACM symposium on user interface software and technology (pp. 75–85). New York: ACM.Google Scholar
  64. Russakovsky, O., Li, L.-J., & Li, F.-F. (2015). Best of both worlds: Human-machine collaboration for object annotation. In IEEE conference on computer vision and pattern recognition (CVPR) (pp. 2121–2131). Boston: IEEE.Google Scholar
  65. Russell, S. H., Hancock, M. P., & McCullough, J. (2007). Benefits of undergraduate research experiences. Science, 316(5824), 548–549.CrossRefGoogle Scholar
  66. Sarsons, H. (2015). Gender differences in recognition for group work. Working Paper, Harvard University.Google Scholar
  67. Shen, H.-W., & Barabási, A.-L. (2014). Collective credit allocation in science. Proceedings of the National Academy of Sciences, 111(34), 12325–12330.CrossRefGoogle Scholar
  68. Silvertown, J. (2009). A new dawn for citizen science. Trends in Ecology & Evolution, 24(9), 467–471.CrossRefGoogle Scholar
  69. Stahl, G. K., Maznevski, M. L., Voigt, A., & Jonsen, K. (2010). Unraveling the effects of cultural diversity in teams: A meta-analysis of research on multicultural work groups. Journal of International Business Studies, 41(4), 690–709.CrossRefGoogle Scholar
  70. Stevens, M., Vitos, M., Altenbuchner, J., Conquest, G., Lewis, J., & Haklay, M. (2014). Taking participatory citizen science to extremes. IEEE Pervasive Computing, 13(2), 20–29.CrossRefGoogle Scholar
  71. Sullivan, B. L., Wood, C. L., Iliff, M. J., Bonney, R. E., Fink, D., & Kelling, S. (2009). eBird: A citizen-based bird observation network in the biological sciences. Biological Conservation, 142(10), 2282–2292.CrossRefGoogle Scholar
  72. Sun, Y., Yintao, Y., & Han, J. (2009). Ranking-based clustering of heterogeneous information networks with star network schema. In Proceedings of the 15th ACM SIGKDD international conference on Knowledge discovery and data mining (pp. 797–806). New York: ACM.Google Scholar
  73. Suzuki, R., Salehi, N., Lam, M. S., Marroquin, J. C., & Bernstein, M. S. (2016). Atelier: Repurposing expert crowdsourcing tasks as micro-internships. In Proceedings of the 2016 CHI conference on human factors in computing systems (pp. 2645–2656). New York: ACM.CrossRefGoogle Scholar
  74. Teevan, J., Iqbal, S. T., & von Veh, C. (2016). Supporting collaborative writing with microtasks. In Proceedings of the 2016 CHI conference on human factors in computing systems (pp. 2657–2668). New York: ACM.CrossRefGoogle Scholar
  75. The Times Higher Education World University Rankings. (2017). https://www.timeshighereducation.com/world-university-rankings
  76. Toegel, G., & Conger, J. A. (2003). 360-degree assessment: Time for reinvention. Academy of Management Learning & Education, 2(3), 297–311.CrossRefGoogle Scholar
  77. Tomlinson, B., Ross, J., Andre, P., Baumer, E., Patterson, D., Corneli, J., Mahaux, M., Nobarany, S., Lazzari, M., Penzenstadler, B., et al. (2012). Massively distributed authorship of academic papers. In Extended abstracts of the 2012 ACM annual conference on human factors in computing systems (pp. 11–20). New York: ACM.CrossRefGoogle Scholar
  78. Tsai, J. L. (2007). Ideal affect: Cultural causes and behavioral consequences. Perspectives on Psychological Science, 2(3), 242–259.CrossRefGoogle Scholar
  79. Valentine, M. A., Retelny, D., To, A., Rahmati, N., Doshi, T., & Bernstein, M. S. (2017). Flash organizations: Crowdsourcing complex work by structuring crowds as organizations. In Proceedings of the 2017 CHI conference on human factors in computing systems (pp. 3523–3537). New York: ACM.CrossRefGoogle Scholar
  80. Veit, A., Wilber, M. J., Vaish, R., Belongie, S. J., Davis, J., Anand, V., Aviral, A., Chakrabarty, P., Chandak, Y., Chaturvedi, S., Devaraj, C., Dhall, A., Dwivedi, U., Gupte, S., Sridhar, S. N., Paga, K., Pahuja, A., Raisinghani, A., Sharma, A., Sharma, S., Sinha, D., Thakkar, N., Bala Vignesh, K., Verma, U., Abhishek, K., Agrawal, A., Aishwarya, A., Bhattacharjee, A., Dhanasekar, S., Gullapalli, V. K., Gupta, S., Chandana, G., Jain, K., Kapur, S., Kasula, M., Kumar, S., Kundaliya, P., Mathur, U., Mishra, A., Mudgal, A., Nadimpalli, A., Nihit, M. S., Periwal, A., Sagar, A., Shah, A., Sharma, V., Sharma, Y., Siddiqui, F., Singh, V., Abhinav, S., Tambwekar, P., Taskin, R., Tripathi, A., & Yadav, A. D. (2015). On optimizing human-machine task assignments. HCOMP 2015 extended abstracts.Google Scholar
  81. Whiting, M. E., Gamage, D., Snehalkumar (Neil) S Gaikwad, Gilbee, A., Goyal, S., Ballav, A., Majeti, D., Chhibber, N., Richmond-Fuller, A., Vargus, F., Sarma, T. S., Chandrakanthan, V., Moura, T., Salih, M. H., Kalejaiye, G. B. T., Ginzberg, A., Mullings, C. A., Dayan, Y., Milland, K., Orefice, H., Regino, J., Parsi, S., Mainali, K., Sehgal, V., Matin, S., Sinha, A., Vaish, R., & Bernstein, M. S. (2017). Crowd guilds: Worker-led reputation and feedback on crowdsourcing platforms. In Proceedings of the 2017 ACM conference on computer supported cooperative work and social computing (CSCW ’17) (pp. 1902–1913). New York: ACM.Google Scholar
  82. Zhang, H., Easterday, M. W., Gerber, E. M., Lewis, D. R., & Maliakal, L. (2017). Agile research studios. In Proceedings of the 2017 ACM conference on computer supported cooperative work and social computing – CSCW ’17 (pp. 220–232). New York: ACM.Google Scholar
  83. Zydney, A. L., Bennett, J. S., Shahid, A., & Bauer, K. W. (2002). Impact of undergraduate research experience in engineering. Journal of Engineering Education, 91, 151–157.CrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  • Rajan Vaish
    • 1
    Email author
  • Snehalkumar (Neil) S. Gaikwad
    • 2
  • Geza Kovacs
    • 1
  • Andreas Veit
    • 3
  • Ranjay Krishna
    • 1
  • Imanol Arrieta Ibarra
    • 1
  • Camelia Simoiu
    • 1
  • Michael Wilber
    • 3
  • Serge Belongie
    • 3
  • Sharad C. Goel
    • 1
  • James Davis
    • 4
  • Michael S. Bernstein
    • 1
  1. 1.Stanford UniversityStanfordUSA
  2. 2.MIT Media LabCambridgeUSA
  3. 3.Cornell TechNew YorkUSA
  4. 4.UC Santa CruzSanta CruzUSA

Personalised recommendations