Further into the Wild: Running Worldwide Trials of Mobile Systems
Many studies of ubiquitous computing systems involve deploying a system to a group of users who will be studied through direct observation, interviews and the gathering of system log data. However, such studies are often limited in the number of participants and duration of the trial, particularly if the researchers are providing the participants with hardware. Apple’s App Store and similar application repositories have become popular with smartphone users, yet few ubiquitous computing studies have yet utilised these distribution mechanisms. We describe our experiences of running a very large scale trial where such a distribution model is used to recruit thousands of users for a mobile system trial that can be run continuously with no constrained end date. We explain how we conducted such a trial, covering issues such as data logging and interviewing users based in several different continents. Benefits and potential shortcomings of running a trial in this way are discussed and we offer guidance on ways to help manage a large and disparate user-base using in-application feedback measures and web-based social networking applications. We describe how, through these methods, we were able to further the development of a piece of ubiquitous computing software through user-informed design on a mass scale.
KeywordsEvaluation Techniques Large Scale Deployment Trial Methods
Unable to display preview. Download preview PDF.
- 2.IDC, Worldwide Converged Mobile Device Market Grows 39.0% Year Over Year in Fourth Quarter, http://www.idc.com/getdoc.jsp?containerId=prUS22196610
- 3.Zhai, S., et al.: Shapewriter on the iPhone: from the laboratory to the real world. In: Proc. ACM CHI Extended Abstracts, pp. 2667–2670 (2009)Google Scholar
- 4.Bell, M., et al.: Interweaving Mobile Games with Everyday Life. In: Proc. ACM CHI, pp. 417–426 (2006)Google Scholar
- 5.Hall, M., et al.: Adapting Ubicomp Software and its Evaluation. In: Proc. ACM Engineering Interactive Computing Systems, pp. 143–148 (2009)Google Scholar
- 8.Chin, A.: Finding Cohesive Subgroups and Relevant Members in the Nokia Friend View Mobile Social Network. In: Proc. Social Computing, pp. 278–283 (2009)Google Scholar
- 10.Crabtree, A., et al.: Supporting Ethnographic Studies of Ubiquitous Computing in the Wild. In: Proc. ACM DIS, pp. 60–69 (2006)Google Scholar
- 12.Carter, S., et al.: Support for Situated Ubicomp Experimentation. In: Proc. ACM CHI, pp. 125–134 (2007)Google Scholar
- 14.Common European Framework of Reference for Languages, http://www.coe.int/T/DG4/Linguistic/CADRE_EN.asp
- 15.Facebook Statistics, http://www.facebook.com/press/info.php?statistics
- 16.FAST, Federation Asks: Do you know what you’re agreeing to?, http://www.fastiis.org/resources/press/id/304/