Abstract
Crowdsourcing systems accomplish large tasks with scale and speed by breaking work down into independent parts. However, many types of complex creative work, such as fiction writing, have remained out of reach for crowds because work is tightly interdependent: changing one part of a story may trigger changes to the overall plot and vice versa. Taking inspiration from how expert authors write, we propose a technique for achieving interdependent complex goals with crowds. With this technique, the crowd loops between reflection, to select a high-level goal, and revision, to decompose that goal into low-level, actionable tasks. We embody this approach in Mechanical Novel, a system that crowdsources short fiction stories on Amazon Mechanical Turk. In a field experiment, Mechanical Novel resulted in higher-quality stories than an iterative crowdsourcing workflow. Our findings suggest that orienting crowd work around high-level goals may enable workers to coordinate their effort to accomplish complex work.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
This story length struck a balance between being long enough to make it difficult to coordinate work and short enough to complete in a reasonable amount of time on Mechanical Turk.
- 2.
The federal minimum wage at the time of this writing was $7.25.
References
Alexander, C., Ishikawa, S., & Silverstein, M. (1977). A pattern language (Vol. 2). Oxford: Oxford University Press.
André, P., Kittur, A., & Dow, S. P. (2014). Crowd synthesis: Extracting categories and clusters from complex data. In Proceedings of the 17th ACM Conference on Computer Supported Cooperative Work & Social Computing, CSCW ’14 (pp. 989–998). New York, NY: ACM. doi:10.1145/2531602.2531653. http://doi.acm.org/10.1145/2531602.2531653
Bayles, D., Orland, T., & Morey, A. (2012). Art & fear. Old Saybrook: Tantor Media, Incorporated.
Bernstein, M. S., Little, G., Miller, R. C., Hartmann, B., Ackerman, M. S., Karger, D. R., Crowell, D., & Panovich, K. (2010). Soylent: A word processor with a crowd inside. In Proceedings of the 23Nd Annual ACM Symposium on User Interface Software and Technology, UIST ’10 (pp. 313–322). New York, NY: ACM. doi:10.1145/1866029.1866078. http://doi.acm.org/10.1145/1866029.1866078
Bigham, J. P., Bernstein, M. S., & Adar, E. (2015). Human-computer interaction and collective intelligence. In T. W. Malone & M. S. Bernstein (Eds.), Collective intelligence handbook. Cambridge: MIT Press.
Burroway, J. (2003). Imaginative writing: The elements of craft. New York: Longman
Chan, J., Dang, S., Kremer, P., Guo, L., & Dow, S. (2014). Ideagens: A social ideation system for guided crowd brainstorming. In Second AAAI Conference on Human Computation and Crowdsourcing.
Chilton, L. B., Little, G., Edge, D., Weld, D. S., & Landay, J. A. (2013). Cascade: Crowdsourcing taxonomy creation. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI ’13 (pp. 1999–2008). New York, NY: ACM. doi:10.1145/2470654.2466265. http://doi.acm.org/10.1145/2470654.2466265
d.school, S. (2016). Design method: I like, i wish, what if. http://dschool.stanford.edu/wp-content/themes/dschool/method-cards/i-like-i-wish-what-if.pdf
Flower, L., & Hayes, J. R. (1981). A cognitive process theory of writing. College Composition and Communication, 32(4), 365–387.
Foldingstory. (2016). http://foldingstory.com/
Hahn, N., Chang, J., Kim, J. E., & Kittur, A. (2016). The knowledge accelerator: Big picture thinking in small pieces. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, CHI ’16 (pp. 2258–2270). New York, NY: ACM. doi:10.1145/2858036.2858364. http://doi.acm.org/10.1145/2858036.2858364
Hill, B. M., & Monroy-Hernández, A. (2013). The cost of collaboration for code and art: Evidence from a remixing community. In Proceedings of the 2013 Conference on Computer Supported Cooperative Work, CSCW ’13 (pp. 1035–1046). New York, NY: ACM. doi:10.1145/2441776.2441893. http://doi.acm.org/10.1145/2441776.2441893
Jansson, D. G., & Smith, S. M. (1991). Design fixation. Design Studies, 12(1), 3–11.
Kim, H. C. E., & Eklundh, K. S. (2001). Reviewing practices in collaborative writing. Computer Supported Cooperative Work, 10(2), 247–259. doi:10.1023/A:1011229212323. http://dx.doi.org/10.1023/A:1011229212323
Kim, J., Cheng, J., & Bernstein, M. S. (2014). Ensemble: Exploring complementary strengths of leaders and crowds in creative collaboration. In Proceedings of the 17th ACM Conference on Computer Supported Cooperative Work & Social Computing, CSCW ’14 (pp. 745–755). New York, NY: ACM. doi:10.1145/2531602.2531638. http://doi.acm.org/10.1145/2531602.2531638
Kim, J., & Monroy-Hernández, A. (2016). Storia: Summarizing social media content based on narrative theory using crowdsourcing. In Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social Computing, CSCW ’16 (pp. 1018–1027). New York: ACM. doi:10.1145/2818048.2820072. http://doi.acm.org/10.1145/2818048.2820072
Kittur, A., Smus, B., Khamkar, S., & Kraut, R. E. (2011). Crowdforge: Crowdsourcing complex work. In Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology, UIST ’11 (pp. 43–52). New York, NY: ACM. doi:10.1145/2047196.2047202. http://doi.acm.org/10.1145/2047196.2047202
Koblin, A. (2010). The Johnny cash project. http://www.thejohnnycashproject.com/
Kulkarni, A., Can, M., & Hartmann, B. (2012). Collaboratively crowdsourcing workflows with turkomatic. In Proceedings of the ACM 2012 Conference on Computer Supported Cooperative Work, CSCW ’12 (pp. 1003–1012). New York, NY: ACM. doi:10.1145/2145204.2145354. http://doi.acm.org/10.1145/2145204.2145354
Lasecki, W., Miller, C., Sadilek, A., Abumoussa, A., Borrello, D., Kushalnagar, R., & Bigham, J. (2012). Real-time captioning by groups of non-experts. In Proceedings of the 25th Annual ACM Symposium on User Interface Software and Technology, UIST ’12 (pp. 23–34). New York, NY: ACM. doi:10.1145/2380116.2380122. http://doi.acm.org/10.1145/2380116.2380122
Lasecki, W. S., Kim, J., Rafter, N., Sen, O., Bigham, J. P., & Bernstein, M. S. (2015). Apparition: Crowdsourced user interfaces that come to life as you sketch them. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems, CHI ’15 (pp. 1925–1934). New York, NY: ACM. doi:10.1145/2702123.2702565. http://doi.acm.org/10.1145/2702123.2702565
Lasecki, W. S., Wesley, R., Nichols, J., Kulkarni, A., Allen, J. F., & Bigham, J. P. (2013). Chorus: A crowd-powered conversational assistant. In Proceedings of the 26th Annual ACM Symposium on User Interface Software and Technology, UIST ’13 (pp. 151–162). New York, NY: ACM. doi:10.1145/2501988.2502057. http://doi.acm.org/10.1145/2501988.2502057
Little, G., Chilton, L. B., Goldman, M., & Miller, R. C. (2010a). Exploring iterative and parallel human computation processes. In Proceedings of the ACM SIGKDD Workshop on Human Computation, HCOMP ’10 (pp. 68–76). New York, NY: ACM. doi:10.1145/1837885.1837907. http://doi.acm.org/10.1145/1837885.1837907
Little, G., Chilton, L. B., Goldman, M., & Miller, R. C. (2010b). Turkit: Human computation algorithms on mechanical turk. In Proceedings of the 23nd Annual ACM Symposium on User Interface Software and Technology, UIST ’10 (pp. 57–66). New York, NY: ACM. doi:10.1145/1866029.1866040. http://doi.acm.org/10.1145/1866029.1866040
Luther, K., Fiesler, C., & Bruckman, A. (2013). Redistributing leadership in online creative collaboration. In Proceedings of the 2013 Conference on Computer Supported Cooperative Work, CSCW ’13 (pp. 1007–1022). New York, NY: ACM. doi:10.1145/2441776.2441891. http://doi.acm.org/10.1145/2441776.2441891
Mason, B., & Thomas, S. (2008). A million penguins research report. Leicester: Institute of Creative Technologies, De Montfort University.
Nebeling, M., To, A., Guo, A., de Freitas, A. A., Teevan, J., Dow, S. P., & Bigham, J. P. (2016). Wearwrite: Crowd-assisted writing from smart watches. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, CHI ’16 (pp. 3834–3846). New York, NY: ACM. doi:10.1145/2858036.2858169. http://doi.acm.org/10.1145/2858036.2858169
Noël, S., & Robert, J. M. (2004). Empirical study on collaborative writing: What do co-authors do, use, and like? Computer Supported Cooperative Work, 13(1), 63–89. doi:10.1023/B:COSU.0000014876.96003.be. http://dx.doi.org/10.1023/B:COSU.0000014876.96003.be
Pecher, D., & Zwaan, R. A. (2005). Grounding cognition: The role of perception and action in memory, language, and thinking. Cambridge: Cambridge University Press.
Retelny, D., Robaszkiewicz, S., To, A., Lasecki, W. S., Patel, J., Rahmati, N., Doshi, T., Valentine, M., & Bernstein, M. S. (2014). Expert crowdsourcing with flash teams. In Proceedings of the 27th Annual ACM Symposium on User Interface Software and Technology, UIST ’14 (pp. 75–85). New York, NY: ACM. doi:10.1145/2642918.2647409. http://doi.acm.org/10.1145/2642918.2647409
Schön, D. A. (1983). The reflective practitioner. London: Temple Smith.
Sharples, M. (1999). An account of writing as creative design. Abingdon: Psychology Press.
Simpson, R., Page, K. R., & De Roure, D. (2014). Zooniverse: Observing the world’s largest citizen science platform. In Proceedings of the 23rd International Conference on World Wide Web, WWW ’14 Companion (pp. 1049–1054). New York, NY: ACM. doi:10.1145/2567948.2579215. http://doi.acm.org/10.1145/2567948.2579215
Teevan, J., Iqbal, S. T., & von Veh, C. (2016). Supporting collaborative writing with microtasks. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, CHI ’16 (pp. 2657–2668). New York, NY: ACM. doi:10.1145/2858036.2858108. http://doi.acm.org/10.1145/2858036.2858108
Thompson, J. D. (1967). Organizations in action: Social science bases of administrative theory. Piscataway: Transaction Publishers.
Verroios, V., & Bernstein, M. S. (2014). Context trees: Crowdsourcing global understanding from local views. In Proceedings of the Second AAAI Conference on Human Computation and Crowdsourcing. http://ilpubs.stanford.edu:8090/1105/
WeAreDynamo: Guidelines for academic requesters. http://wiki.wearedynamo.org/index.php/Guidelines_for_Academic_Requesters Accessed May 15, 2014.
Xu, A., Huang, S. W., & Bailey, B. (2014). Voyant: Generating structured feedback on visual designs using a crowd of non-experts. In Proceedings of the 17th ACM Conference on Computer Supported Cooperative Work & Social Computing, CSCW ’14 (pp. 1433–1444). New York, NY: ACM. doi:10.1145/2531602.2531604. http://doi.acm.org/10.1145/2531602.2531604
Yu, L., & Nickerson, J. V. (2011). Cooks or cobblers?: Crowd creativity through combination. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI ’11 (pp. 1393–1402). New York, NY: ACM. doi:10.1145/1978942.1979147. http://doi.acm.org/10.1145/1978942.1979147
Zhang, H., Law, E., Miller, R., Gajos, K., Parkes, D., & Horvitz, E. (2012). Human computation tasks with global constraints. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI ’12 (pp. 217–226). New York, NY: ACM. doi:10.1145/2207676.2207708. http://doi.acm.org/10.1145/2207676.2207708
Acknowledgements
We would like to thank Mechanical Turk workers and study participants for their time and valuable feedback. Thanks also to our colleagues who helped test early prototypes of Mechanical Novel. Special thanks to Kylie Jue for her help developing the Mechanical Novel system. This material is based upon work supported by the NSF under Grant No. DGE-114747 and Grant No. IIS-1351131 and by the Hasso Plattner Institute Design Thinking Research Program.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Appendix
Appendix
Below is one of the stories that workers wrote using Mechanical Novel. When Kaley was five, she was given a very special gift by her grandmother, a beautiful blue stuffed elephant. This wasn’t just any stuffed elephant—it was a handmade stuffed elephant created for Kaley’s mother when she was just a girl, that she had loved dearly. Her mother had passed away when Kaley was a baby and so Kaley was raised by her grandma. The grandmother did her best, but there was always something missing, which made this elephant extra special because it made Kaley feel like she still had part of her mother with her, even though she knew that was crazy. Kaley not only loved her blue elephant because it belonged to her mother, but the elephant also became her best friend. The elephant was always the guest of honor at her tea parties and always slept by her at night and Kaley always felt safe as long as the elephant was with her.
Kaley often asked her grandma to tell her stories about her mother. She would sit on her lap and hold her elephant while grams told her lovely things about her mom. She loved those precious moments and wanted to ask grandma for a story later on. She couldn’t wait! Holding her special elephant and hearing these moments from her mother’s life was comforting to her.
What Kaley didn’t know was that the elephant was indeed a very special elephant, special beyond her wildest imagination. Because Kaley’s mom had loved the elephant so dearly, a part of her had lived on through the elephant. On the morning of Kaley’s sixth birthday she woke as the sun danced across her bed and was excited for her party that day. “Elephant!” she exclaimed, “Today is my birthday and we shall have guests, and cake and presents!” She turned to hug the blue elephant in excitement, but the elephant was gone. “How strange.” she thought to herself as she looked to the side of the bed. No elephant there. She climbed down to look under the bed for the elephant but no elephant there either. She sat back on her heels as she was puzzled at where her elephant could be. “How could it have just disappeared?” she thought to herself. Kaley was soon to find out how and just how special her blue elephant really was.
It’s not like blue elephants just get up and walk away on their own…. or do they? Kaley’s blue elephant wasn’t like other blue elephants, that’s why she always wrote his name in capital letters in her diary and when she wrote short stories at school. “Blue Elephant”, just like that. That was his name, after all! Kaley’s elephant was blue with a long trunk. She got it from her grandma when she was young and it has been with her ever since. She got it on her 5th birthday as a gift. Maybe he did just get up and walk away. I wouldn’t be a bit surprised! I better go look for him right now!
Kaley started her search for the Blue Elephant. First, she searched her room looking in every nook and cranny. No Blue Elephant. After a very long day of searching and not finding Blue Elephant, Kaley started to cry. Kaley’s mother had an idea…. She gave Kaley some peanuts to put out to help catch Blue Elephant. Finally Kaley fell asleep for the night. When she woke up in the morning, Blue Elephant was in her bed with a stash of peanuts.
The girl called for the elephant as loudly as she could. He must have heard her, his big ears make it possible to hear from miles away. If he were trapped or something he would surely be able to send a reply with his giant trunk. She wondered where he could be and wandered down the road calling loudly for him. Every few steps she would sit still and listen for him. Then, she thought she heard a muffled reply and put her ear to the ground. She felt the soft thump of an elephant from a far away distance.
Sure enough, Blue was floating gleefully in the pool spraying water triumphantly from his trunk. Kaley could scarcely belief her eyes, but the glee of her imagination took hold and she yelled in joy, “Blue! Blue! Is that you?” At the sound of her voice, the little elephant turned his trunk and blew water all over her. Leaving her soaking wet and giggling at her silly little friend.
“How did this happen?” asked Kaley. The blue elephant was delighted to answer her question. “You see, Kaley, it was through your love and adoration that I was able to come to life! If it weren’t for you I wouldn’t be here. Remember that wish you made the day before since? Well, it came true! The spirit of your grandmother lives on, in me. She wanted nothing but for you to be happy. Because of your love I’m here and will answer anything you ask” Shocked, Kaley took a step back and assessed the situation. “Well, I suppose this wasn’t such a bad wish!” She thought about what she would ask but really all she wanted was to tell her grandma that she missed her “I miss you grandma, you were gone too soon…” “ Your grandmother would be happy to hear that Kaley and please tell your mother that she loved her no matter how things turned out”. “ Elephant? Are you going to stay?” “ I’m afraid not. Grandma’s spirit has given me only a temporary time with you and it’s just about to expire” Just like that the dol started to glow and landed in Kaley’s hand. Kaley hugged the doll. A doll that she will forever cherish.
Rights and permissions
Copyright information
© 2018 Springer International Publishing AG
About this chapter
Cite this chapter
Kim, J., Sterman, S., Cohen, A.A.B., Bernstein, M.S. (2018). Mechanical Novel: Crowdsourcing Complex Work Through Reflection and Revision. In: Plattner, H., Meinel, C., Leifer, L. (eds) Design Thinking Research. Understanding Innovation. Springer, Cham. https://doi.org/10.1007/978-3-319-60967-6_5
Download citation
DOI: https://doi.org/10.1007/978-3-319-60967-6_5
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-60966-9
Online ISBN: 978-3-319-60967-6
eBook Packages: Business and ManagementBusiness and Management (R0)