smid: A Black-Box Program Driver
- 490 Downloads
We aim to perform dynamic analysis at large scale and across a wide range of programs. A key problem to overcome is driving interactive programs effectively and efficiently. To address this problem, we developed smid—an open-source tool that autonomously interacts with computer programs, based on a specification of which user interactions (key presses, mouse events) are valid. Users can define the space of valid user interactions as a state machine in the smid language. smid can then generate ‘sensible’ program runs (sequences of user interactions), which it sends to the target program. Runs can be saved and played back, facilitating the reproduction of bugs. We have used smid to reproduce and help explain a bug in the cmus music player. It is possible to use smid to drive a wide variety of desktop programs, including those with a graphical, console, or even non-interactive user interface.
KeywordsState Machine User Interaction Target Program Window System Desktop Application
Multiple efforts at automated static analysis of large software repositories exist, and have had a measurable impact on software quality. Notably, the experiment described in  has resulted in hundreds of bug reports, many of which have since been closed as fixed; and the Debile1 effort has provided developers with the results of running several static analysers through a uniform interface.
Automation. We require the ability to automatically drive the user interfaces of computer programs.
Wide Applicability. We require that our tool can be used to drive a wide variety of software, regardless of the high-level user interface toolkit used.
Reproducibility. If a certain sequence of user inputs gives rise to an interesting behaviour (e.g. a crash), we require the ability to ‘play back’ that sequence in order to reproduce the behaviour.
Realism. We do not wish to spam target programs with random inputs, but would like to be able to exercise target programs with only those sequences of user inputs that a real user might issue.
Conciseness. On the other hand, we do not wish to define every possible use case as a set of user interaction ‘scripts’, but would rather have a concise definition of the desired user interactions with the target program.
We have implemented smid (the state machine interface driver)—an open-source2 tool that autonomously interacts with desktop computer programs, driving them with user input in the same way that a real user might. In this paper, we outline the aims and motivation of the tool in Sect. 1; describe smid ’s input language in Sect. 2; and describe the tool’s usage and how we used it to reproduce a bug in Sect. 3. A video demo of the tool can be viewed on YouTube3.
With smid, the space of valid interactions with a program is understood as a state machine—that is, a directed graph whose nodes are states and whose transitions are sequences of user interactions. A state is a point during the program’s execution where a certain set of transitions are possible; these are often referred to as ‘modes’ in the user interaction literature .
Example: In a graphical web browser such as Firefox or Chromium, the main browser window, as well as each of the auxiliary windows (download window, print dialog, etc.) can be understood as states. Transitions (key presses and mouse events) may cause the state to remain the same (e.g. when pressing Ctrl+R to reload a web page), or change the state (e.g. clicking on the ‘Options’ menu item changes the state to the Options window). Different views in the same window may also be understood as separate states.
To use smid, users describe such a state machine by writing a file in the smid language. The file is parsed by the smid tool, which outputs a single ‘run’ (a finite path through the state machine, from the initial to a final state). smid then autonomously ‘drives’ the program by ‘playing back’ this run—that is, by sending the sequence of user interactions to the target program. By playing back a large number of runs, the target program can be rigorously tested.
1.2 Comparison with Existing Tools
Tools that we have explored, both in academia and in the wider software development community, fell short of one or more of the requirements listed above.
In both academia and industry, the majority of black-box driving tools are only able to interact with programs using one particular user interface toolkit (UIT). The tools Selenium , Mozmill , Watir  and Canoo  can only be used to autonomously interact with web applications. EXSYST  is a Java GUI automator, although its search-based technique is not limited to Java programs. A tool that does not have the single-UIT limitation is Jubula , which is able to drive applications using one of several web and desktop UITs.
Jubula and the web application drivers mentioned above work by executing a ‘script’ of user interactions, which corresponds to a single use case. To test the target program on all use cases, one must write a large number of scripts; changes in the underlying program then require updates to all affected scripts. We find this method to be fragile, and mention how smid negates the need for redundant specification in Sect. 3. There has been some work in academia on automatically determining the behaviour of the target program. Amalfitano et al. [1, 2] describe techniques for determining the behaviour of Android applications and web applications, respectively. A formal approach to designing, specifying, and subsequently generating tests for graphical user interfaces is described in . There has also been work to automatically understand user interface behaviour using techniques from artificial intelligence  or machine learning . Nevertheless, these techniques have each been implemented for only a single UIT. One possibility for future work is to use the Linux Desktop Testing Project  (LDTP) to provide the highly semantic information that is needed for these ‘behaviour-learning’ tools, for a wide range of UITs and platforms.
In contrast, smid allows the specification of the set of user interactions that are sensible for the target program using the language described in Sect. 2, without burdening the user with writing interaction ‘scripts’ and repeatedly specifying common interaction sequences. The fact that smid sends events directly to the X Window System—which underlies all graphical user interface toolkits on desktop Linux and BSD operating systems—means that smid is able to drive all interactive desktop applications, as well as console-based applications (through interaction with a terminal emulator), that run on an X-based operating system. The smid language itself is UIT-agnostic, and we hope to use LDTP in the future in order to provide smid users with interactions that are more semantic than raw interactions with X—for example, by specifying graphical widgets by name rather than by coordinate.
1.3 Scope and Limitations
smid is aimed at running experiments on the large body of software supplied with a Linux distribution. Accordingly, while the language described in Sect. 2 is UIT-agnostic, our implementation of smid targets desktop applications; we did not attempt to implement driving of touch- or voice-based user interfaces. The back-end to our tool is xdotool , an API for sending events to an X server, but we expect that smid can be modified to use a different back-end (like LDTP) in order to make it usable on other systems. While smid is used to drive target programs toward error states, smid does not implement any error detection itself; we leave application-specific error-detection to the user.
2 The SMIDsmid Language
The smid language is used to describe a user interface as a state machine. Files written in the smid language consist mostly of transition statements; in this section, we describe the various forms that transitions can take, as well as other features of the smid language. The most up-to-date guide to the language is the reference manual4 found on the smid web site.
3 SMIDsmid Usage and Case Study
Generating a Run. smid can output a list of user actions, called a ‘run,’ by accumulating the actions along a finite-length walk of the state machine.
Playing Back a Run. smid can read a run and sends the specified interactions to the target program. Thus, given a run, smid can autonomously interact with the target program in the same way a real user might.
In this section, we describe using smid to pinpoint crashes in the target program.
We wrote a smid file (available on GitHub5) for the cmus 6 console music player. This file had 25 transitions, which smid expanded into a 103-transition state machine. Hence, we find that the smid language allows us to specify a wide range of behaviour in a concise format and with little redundancy.
We used this smid file to reproduce a reported7 segmentation fault. The bug report suggested that the bug was triggered when playing MP4-encoded media files. We set up a large library of audio and video containers, and a smid specification designed to hone in on the reported bug. smid caused cmus to browse to one of the media files, seek to a random point in the file, and play the data for several seconds. Using this setup, smid triggered the segfault in many of the several hundred runs that we ran.
By logging cmus using the SystemTap  instrumentation framework while running it with smid, we were able to discover several scenarios under which this bug was triggered, but which were not described in the original bug report. This demonstrates the utility of our approach—namely, sending a diverse range of inputs to the target program, from the space of ‘sensible’ user interactions.
The smid language, described in Sect. 2, is UIT and platform agnostic. The current implementation of the smid tool uses this fact to drive programs by sending user interactions directly to the underlying window system, rather than to a specific UIT. This means that we are able to drive the large variety of applications that can render a window under the X Window System.
Existing approaches either try to learn the state machine—tying them down to particular UITs, or drive the interface using a script—an approach which is not scaleable. The case study in Sect. 3 shows the value of our approach: by specifying all reasonable behaviours of the target program, we were able to quickly hone in on a bug without spamming the target program with unrealistic inputs.
We thank Carsten Fush and Tyler Sorensen, as well as several anonymous reviewers, for constructive feedback.
- 1.Amalfitano, D., Fasolino, A.R., Tramontana, P.: Reverse engineering finite state machines from rich internet applications. In: WCRE (2008)Google Scholar
- 2.Amalfitano, D., Fasolino, A.R., Tramontana, P., Carmine, S.D., Memon, A.M.: Using GUI ripping for automated testing of Android applications. In: ASE (2012)Google Scholar
- 3.Berstel, J., Crespi-Reghizzi, S., Roussel, G., Pietro, P.S.: A scalable formal method for design and automatic checking of user interfaces. In: ICSE (2001)Google Scholar
- 4.Chen, E., LDTP contributors: Linux desktop testing project. http://ldtp.freedesktop.org/
- 5.Dan, H., Harman, M., Krinke, J., Li, L., Marginean, A., Wu, F.: Pidgin crasher: searching for minimised crashing GUI event sequences. In: Goues, C., Yoo, S. (eds.) SSBSE 2014. LNCS, vol. 8636, pp. 253–258. Springer, Heidelberg (2014)Google Scholar
- 6.Eigler, F.C., Stone, D., Stone, J., Wielaard, M., SystemTap contributors: System-Tap. https://sourceware.org/systemtap/
- 7.Gross, F., Fraser, G., Zeller, A.: EXSYST: search-based GUI testing. In: ICSE (2012)Google Scholar
- 8.Huber, M., Schlichting, M., Canoo contributors: Canoo WebTest. http://webtest.canoo.com/
- 9.Kroening, D., Tautschnig, M.: Automating software analysis at large scale. In: Hliněný, P., Dvořák, Z., Jaroš, J., Kofroň, J., Kořenek, J., Matula, P., Pala, K. (eds.) MEMICS 2014. LNCS, vol. 8934, pp. 30–39. Springer, Heidelberg (2014)Google Scholar
- 10.Memon, A.M., Pollack, M.E., Soffa, M.L.: Using a goal-driven approach to generate test cases for GUIs. In: ICSE (1999)Google Scholar
- 11.Pertman, J., McHowan, H., Rodionov, A.: Watir. http://watir.com/
- 12.Raskin, J.: The Humane Interface: New Directions for Designing Interactive Systems. ACM Press/Addison-Wesley Publishing Co., New York, NY (2000)Google Scholar
- 13.Sissel, J.: xdotool–fake keyboard/mouse input, window management, and more. http://www.semicomplete.com/projects/xdotool/
- 14.Skupin, H., Hammel, J., Rogers, M., Mozmill contributors: Mozmill. https://developer.mozilla.org/en-US/docs/Mozilla/Projects/Mozmill
- 15.Stewart, S., Selenium contributors: Selinium WebDriver. http://www.seleniumhq.org/projects/webdriver/
- 16.Tiede, M., Struckmann, S., Mueller, M., Jubula contributors: The Jubula functional testing tool. http://www.eclipse.org/jubula/