Exception handling bug hazards in Android
- 466 Downloads
Adequate handling of exceptions has proven difficult for many software engineers. Mobile app developers in particular, have to cope with compatibility, middleware, memory constraints, and battery restrictions. The goal of this paper is to obtain a thorough understanding of common exception handling bug hazards that app developers face. To that end, we first provide a detailed empirical study of over 6,000 Java exception stack traces we extracted from over 600 open source Android projects. Key insights from this study include common causes for system crashes, and common chains of wrappings between checked and unchecked exceptions. Furthermore, we provide a survey with 71 developers involved in at least one of the projects analyzed. The results corroborate the stack trace findings, and indicate that developers are unaware of frequently occurring undocumented exception handling behavior. Overall, the findings of our study call for tool support to help developers understand their own and third party exception handling and wrapping logic.
KeywordsException handling Android development Repository mining Exploratory survey
The number of mobile apps is increasing at a daily rate. If on the one hand they extend phones’ capabilities far beyond the basic calls, on the other hand they have to cope with an increasing number of exceptional conditions (e.g., faults in underlying middleware or hardware; compatibility issues McDonnell et al. 2013; memory and battery restrictions; noisy external resources Zhang and Elbaum 2012).
Therefore, mechanisms for exception detection and handling are not an optional add-on but a fundamental part of such apps. The exception handling mechanism (Goodenough 1975), embedded in many mainstream programming languages, such as Java, C++ and C#, is one of the most used techniques for detecting and recovering from such exceptional conditions. In this paper we will be concerned with exception handling in Android apps, which reuses Java’s exception handling model.
However, such mechanisms are more often than not the least understood and tested parts of the system (Miller and Tripathi 1997; Robillard and Murphy 2000; Shah et al. 2010; Garcia et al. 2007, 2001; Cabral and Marques 2007; Coelho et al. 2011; Coelho et al. 2014). As a consequence they may inadvertently negatively affect the system: exception-related code may introduce failures such as uncaught exceptions (Jo et al. 2004; Zhang and Elbaum 2012) – which can lead to system crashes, making the system even less robust (Coelho et al. 2011).
In Java, when an application fails due to an uncaught exception, it automatically terminates, while the system prints a stack trace to the console, or to a log file (Gosling 2000). A typical Java stack trace consists of the fully qualified name of the thrown exception and the ordered list of methods that were active on the call stack before the exception occurred (Gosling 2000; Bloch 2008). When available, the exception stack traces provide a useful source of information about system crashes (Bettenburg et al. 2008a) which can enable different kinds of post-mortem analysis and support: debugging (Schröter et al. 2010), bug classification and clustering (Wang et al. 2013; Kim et al. 2011; Dhaliwal et al. 2011), automated bug fixing (Sinha et al. 2009) and fault-proneness prediction models (Kim et al. 2013).
This work is conducted in two phases. First, a mining study performs a post mortem analysis of the exception stack traces included in issues reported on Android projects hosted on GitHub and Google Code. The goal of this study is to investigate whether the reported exception stack traces can reveal common bug hazards in the exception-related code. A bug hazard (Binder 2000) is a circumstance that increases the chance of a bug being present in the software. An example of a bug hazard can be a characteristic of the exception-related code which can increase the likelihood of introducing the aforementioned uncaught exceptions. Second, we conducted an exploratory survey with the Android developers involved in the mined projects to assess their perspective about the exception handling bug hazards found.
Cross-type exception wrappings, such as an OutOfMemoryError wrapped in a checked exception. Trying to handle an instance of OutOfMemoryError “hidden” in a checked exception may bring the program to an unpredictable state. Such wrappings suggest that, when (mis)applied, the exception wrapping can make the exception-related code more complex and negatively impact the application robustness.
Undocumented runtime exceptions raised by the Android platform (35 methods) and third-party libraries (44 methods) – which correspond to 4.4 % of the reported exception stack traces. In the absence of the “exception specification” of third-party code, it is difficult or even infeasible for the developer to protect the code against “unforeseen” exceptions. Since in such cases the client usually does not have access to the source code, such undocumented exceptions may remain uncaught and lead to system crashes.
Undocumented checked exceptions signaled by native C code. Some flows contained a checked exception signaled by native C code invoked by the Android Platform, yet this exception was not declared in the Java Native Interface invoking it. This can lead to uncaught exceptions that are difficult to debug.
A multitude of programming mistakes – approximately 52 % of the reported stack traces can be attributed to programming mistakes. In particular, 27.71 % of all stack traces contained a java.lang.NullPointerException as their root cause.
The exploratory survey was conducted with 71 Android developers involved in one or more of the GitHub Android projects whose issues were mined. This survey reveals that only few developers (3 % of respondents) knew about the undocumented checked exceptions signaled by native C code of the Android platform. Moreover, most of the developers recognized that cross-type exception wrappings may negatively impact the application robustness. The uncaught exceptions due to errors in programming logic, e.g. the NullPointerException, were identified by most developers (68 %) as the first or second main cause of application crashes.
The high prevalence of NullPointerExceptions found in the mining study and confirmed in the exploratory survey is in line with findings of earlier research (Kim et al. 2013; Fraser and Arcuri 2013; Csallner and Smaragdakis 2004), as are the undocumented runtime exceptions signaled by the Android Platform (Kechagia and Spinellis 2014).
Some of the findings of our mining study emphasize the impact of these bug hazards on the application robustness by mining a different source of information as the ones used in previous works. The present work mined issues created by developers on GitHub and Google Code, while previous research analyzed crash reports and automated test reports. Furthermore, our work points to bug hazards that were not detected by previous research (i.e., cross-type wrappings, undocumented checked exceptions and undocumented runtime exceptions thrown by third-party libraries) which represent new threats to application robustness. Moreover, we perform the first exploratory survey study whose goal was to assess developers’ perspective regarding a set of exception handling bug hazards as well as how developers deal with exceptions and prevent crashes while developing.
Our findings point to threats not only to the development of robust Android apps, but also to the development of any robust Java-based system. Hence, the study results are relevant to Android and Java developers who may underestimate the effect of such bug hazards on the application robustness, and who have to face the difficulty of preventing them. Moreover, such bug hazards call for improvements of languages (e.g. to prevent null pointer dereferences) and tools to better support exception handling in Android and Java environments.
The remainder of this paper is organized as follows. Section 2 provides the necessary background on the Android platform and the Java exception model. Section 3 presents the mining study design, describes the ExceptionMiner tool we developed to conduct our study, and reports the study results. Section 4 details the exploratory survey. Section 5 provides a discussion of the wider implications of our results, presents the threats to validity associated with the mining study and discusses the limitations of the survey-based study, and points to the replication. Finally Section 6 describes related work, and Section 7 concludes the paper and outlines directions for future work.
2.1 The Android Platform
Android is an open source platform for mobile devices based on the Linux kernel. Android also comprises (i) a set of native libraries written in C/C++ (e.g., WebKit, OpenGL, FreeType, SQLite, Media, C runtime library) to fulfill a wide range of functions including graphics drawing, SSL communication, SQLite database management, audio and video playback etc; (ii) a set of Java Core Libraries including a subset of the Java standard libraries and various wrappers to access the set of C/C++ native libraries using the Java Native Interface (JNI); (iii) the Dalvik runtime environment, which was specifically designed to deal with the resource constraints of a mobile device; and (iv) the Application Framework which provides higher-level APIs to the applications running on the platform.
2.2 Exception Model in Java
In Java, exceptions are represented according to a class hierarchy, in which every exception is an instance of the Throwable class, and can be of three kinds: the checked exceptions (extends Exception), the runtime exceptions (extends RuntimeException) and errors (extends Error) (Gosling 2000). Checked exceptions received their name because they must be declared in the method’s exception interface (i.e., the list of exceptions that a method might raise during its execution) and the compiler statically checks if appropriate handlers are provided within the system. Both runtime exceptions and errors are also known as “unchecked exceptions”, as they do not need to be specified in the method exception interface and do not trigger any compile time checking.
By convention, instances of Error represent unrecoverable conditions which usually result from failures detected by the Java Virtual Machine due to resource limitations, such as OutOfMemoryError. Normally these cannot be handled inside the application. Instances of RuntimeException are implicitly thrown by the Java runtime environment when a program violates the semantic constraints of the Java programming language (e.g., out-of-bounds array index, divide-by-zero error, null pointer references). Some programming languages react to such errors by immediately terminating the program, while other languages, such as C++, let the program continue its execution in some situations such as the out-of-bounds array index. According to the Java Specification (Gosling 2000) programs are not expected to handle such runtime exceptions signaled by the runtime environment.
User-defined exceptions can be either checked or unchecked, by extending either Exception or RuntimeException. There is a long-lasting debate about the pros and cons of both approaches (The Java tutorial. Unchecked exceptions: The controversy ??; Stackoverflow Q&A. Java: checked vs unchecked exception explanation ??; Jenkov Tutorials. Checked or Unchecked Exceptions ??). Section 2.3 presents a set of best practices related to each of them.
In Java, once an exception is thrown, the runtime environment looks for the nearest enclosing exception handler (Java’s try-catch block), and unwinds the execution stack if necessary. This search for the handler on the invocation stack aims at increasing software reusability, since the invoker of an operation can handle the exception in a wider context (Miller and Tripathi 1997).
2.3 Best Practices
Several general guidelines have been proposed on how to use Java exceptions (Mandrioli and Meyer 1992; Gosling 2000; Wirfs-Brock 2006; Bloch 2008). Such guidelines do not advocate any specific exception type, but rather propose ways to effectively use each of them. Based on these, for the purpose of our analysis we compiled the following list of Java exception handling best practices.
I-Checked Exceptions Should be Used to Represent Recoverable Conditions (Mandrioli and Meyer 1992; Gosling 2000; Wirfs-Brock 2006; Bloch 2008)
The developer should use checked exceptions for conditions from which the caller is expected to recover. By confronting the API user with a checked exception, the API designer is forcing the client to handle the exceptional condition. The client can explicitly ignore the exception (swallowing, or converting it to another type) at the expense of the program’s robustness (Gosling 2000).
II-Error Represents an Unrecoverable Condition Which Should not be Handled (Gosling 2000)
Errors should result from failures detected by the runtime environment which indicate resource deficiencies, invariant failures or other conditions, from which the program cannot possibly recover.
III-A Method Should Throw Exceptions that Precisely Define the Exceptional Condition (Gosling 2000; Bloch 2008)
To do so, developers should either try to reuse the exception types already defined in the Java API or they should create a specific exception. Thus, throwing general types such as a pure java.lang.Exception or a java.lang.RuntimeException is considered bad practice.
IV- All Exceptions Explicitly Thrown by Reusable Code Should be Documented (Mandrioli and Meyer 1992; Gosling 2000; Wirfs-Brock 2006; Bloch 2008)
For checked exceptions, this is automatically the case. Bloch (2008) furthermore recommends to document explicitly thrown run time exceptions, either using a throws declaration in the signature, or using the @throws tag in the Javadoc. Doing so, in particular for public APIs of libraries or frameworks, makes clients aware of all exceptions possibly thrown, enabling them to design the code to deal with them and use the API effectively (Robillard and Murphy 2000; Wirfs-Brock 2006).
3 The Repository Mining Study
Our mining study was guided by a general research question: RQ 1: Can the information available in exception stack traces reveal exception handlingbug hazardsin both the Android applications and framework? As mentioned before, in this context bug hazards are the characteristics of exception-related code that favor the introduction of failures such as uncaught exceptions.
Our study focused on the exception stack traces contained in issues of Android projects (hosted on GitHub and Google Code). To support our investigation, we developed a tool called ExceptionMiner (Section 3.3) which extracts the exception stack traces embedded in issues and combines stack trace information with source code and bytecode analysis. Moreover, we use manual inspection to augment the understanding of stack traces and support further discussions and insights (Section 3.4). In this study we explore the domain quantitatively and highlight interesting cases by exploring cases qualitatively.
Our study focuses on open-source apps, since the information needed to perform our study cannot be retrieved from commercial apps, whose issue report systems and source code are generally not publicly available. Open source Android apps have also been the target of other research (Linares-Vásquez et al. 2013; Ruiz et al. 2012) addressing reuse and API stability.
3.1 Android Apps on GitHub
This study uses the dataset provided by the GHTorrent project (Gousios 2013), an off-line mirror of the data offered through the GitHub API. To identify Android projects, we performed a case insensitive search for the term “android” in the repositories’ names and short descriptions. Up to 23 February 2014, when we queried GHTorrent, this resulted in 2,542 repositories. Running the ExceptionMiner tool on this set we observed that 589 projects had at least one issue containing a stack trace.
Then we performed a further clean up, inspecting the site of every Android project reporting at least one stack trace, to make sure that they represented real mobile apps. During this clean up 107 apps were removed because they were either example projects (i.e., toy projects) or tools to support Android development (e.g. Selendroid, Roboeletric – tools to support the testing of Android apps). The filtered set consisted of 482 apps. This set of 482 projects contained a total of 31,592 issues from which 4,042 exception stack traces were extracted.
Labels on issues including exception stack traces
3.2 Android Apps in Google Code
Google Code contains widely used open-source Android apps (e.g. K9Mail1). However, differently from GitHub, Google Code does not provide an API to access the information related to hosted projects.2 To overcome this limitation we needed to implement a Web Crawler (incorporated in the ExceptionMiner tool described next) that navigates the web interface of Google Code projects extracting all issues and issue comments and storing them in a relational database for later analysis. To identify Android projects on Google Code, we performed a similar heuristic: we performed a case insensitive search (on the Google Code search interface) for the term “android”. In January 2014, when we queried Google Code, this resulted in a list of 788 projects. This list comprised the seeds sent to our Crawler.
The Crawler retrieved all issues and comments for these projects. From this set, 724 projects defined at least 1 issue. Running the ExceptionMiner tool on this set we observed that 183 projects had at least one issue containing an exception stack trace. Then we performed further clean up (similar to the one described previously) inspecting the site of each project. As a result we could identify 157 Android projects. This set contained 127,456 issues in total, from which 1,963 exception stack traces were extracted. Table 1 illustrates the occurrences of different labels on the issues including exception stack traces. Differently from GitHub, on Google Code most of the issues were labeled as “Defect”. However, based on the same assumption described for the GitHub repository we considered all issues reporting stack traces (regardless of their labels).
3.3 The ExceptionMiner Tool
The ExceptionMiner is a tool which can connect to different issue repositories, extract issues, mine exception stack traces from them, distill exception stack trace information, and enable the execution of different analyses by combining exception stack trace information with byte code and source code analysis. The main components of ExceptionMiner are the following:
This component enables the connection with issue repositories. In this study two main connectors were created: one which connects to the GHTorrent database, and a Google Code connector which is comprised of a Web Crawler that can traverse the Google Code web interface and extract a project’s issues. Project meta-data and the issues associated with each project are stored in a relational database.
Exception Stack Trace Distiller
This component combines a parser (based on regular expressions) and heuristics able to identify and filter exception names and stack traces inline with text. This component distills the information that composes a stack trace. Some of the attributes extracted from the stack trace are the root exception and its signaler, as well as the exception wrappers and their corresponding signalers. This component also distills fine grained information of each attribute such as the classes and packages associated with them. In contrast to existing issue parsing solutions such as Infozilla, our parser can discover stack traces mixed with log file information.3
Exception Type Analysis
To support a deeper investigation of the stack traces every exception defined in a stack trace needs to be classified according to its type (e.g. Error, checked Exception or RuntimeException). The module responsible for this analysis uses the Design Wizard framework (Brunet et al. 2009) to inspect the bytecode of the exceptions reported in stack traces. It walks up the type hierarchy of a Java exception until it reaches a base exception type. Hence in this study the bytecode analysis was used to discover the type of each mined exception when the jar file of such an exception was available in the project or in a reused Java library. A specific implementation (based on source code analysis) was needed to discover the exception type when the bytecode was not available. With this module we analyzed all exceptions defined in the Android platform (Version 4.4, API level 19), which includes all basic Java exceptions that can be thrown during the app execution, and exceptions thrown by Android core libraries. Moreover, we also analyzed the exceptions reported in stack traces that were defined by applications and third-party libraries (the tool only analyzed the last version available).
Exception Signaler Analysis
Sources of exceptions in Android
If the exception is thrown in a method defined in the Android platform.
If the exception is thrown in a method defined in an Android app.
If the exception is thrown in one of the core libraries reused by Android (e.g., org.apache.harmony, org.w3c.dom, sun.misc, org.apache.http, org.json, org.xml).
If the exception is thrown in a method that was not defined by any of the elements above.
The exceptions are considered to come from libraries if their packages are neither defined within the Android platform, nor in core libraries, nor in the applications. Table 2 summarizes this signaler classification.
3.4 Manual Inspections
In our experiments, the output of the ExceptionMiner tool was manually extended in order to (i) support the identification of packages composing the Android platform, libs and apps analyzed in this study (as described previously); and (ii) identify the type of some exceptions reported in issues that were not automatically identified by the ExceptionMiner tool (because they were defined in previous versions of libraries, apps and Android Platform). When the exception could not be found automatically or manually (because they were defined in a previous version of the app or lib), we classified the exception as “Undefined”. Only 31 exceptions remained undefined, which occurred in 60 different exception stack traces (see Table 5).
3.5 The Mining Study Results
This section presents the results of our study that was guided by the general research question: RQ 1: Can the information available in exception stack traces reveal exception handling bug hazards in both the Android applications and framework? To make the analysis easier, we further refine this question into sub-questions, each one focusing on pieces of information distilled from stack traces, more specifically: (i) the root exceptions (i.e., the exceptions that caused the stack traces); (ii) the exception types (i.e, Checked, Runtime, Error, Throwable) and (iii) the exception wrappings. Hence, this section is centered around the following sub-questions: RQ 1.1: Can the root exceptions reveal bug hazards?; RQ 1.2 Can the exception types reveal bug hazards?; and RQ 1.3 Can the exception wrappings reveal bug hazards?.
RQ 1.1 Can the root exceptions reveal bug harzards?
Root Exceptions occurrences and popularity in analyzed repositories
We can observe that most of the exceptions in this list are implicitly thrown by the runtime environment due to programming mistakes (e.g., out-of-bounds array index, division-by-zero, access to a null reference) or resource limitations (e.g., OutOfMemoryError). From this set the java.lang.NullPointerException was the most reported root cause (27.71 %). If we consider the frequency of NullPointerException across projects, we can observe that 51.96 % of all projects reported at least one exception stack trace in which the NullPointerException was the root cause.
The NullPointerException was mainly signaled inside the application code (50 %) and the Android platform (31.5 %), although we could also find the NullPointerException being signaled by third-party libraries (16.3 %). Regarding reusable code (e.g., libraries and frameworks), there is no consensus whether it is a good or a bad practice to explicitly throw a NullPointerException. Some prefer to encapsulate such an exception in an instance of IllegalArgumentException, while others (Bloch 2008) argue that the NullPointerException makes the cause of the problem explicit and hence can be signaled by an API expecting a non-null argument.
The high prevalence of NullPointerException is aligned with the findings of other research (Kim et al. 2013; Fraser and Arcuri 2013; Csallner and Smaragdakis 2004; Kechagia and Spinellis 2014). For instance, Kechagia and Spinellis showed that the NullPointerException was the most reported exception in the crash reports sent to BugSense4 (a bug report management service for Android applications) (Kechagia and Spinellis 2014). Other research on robustness testing (Maji et al. 2012; Csallner and Smaragdakis 2004) shows that most of the automatically detected bugs were due to NullPointerException and exceptions implicitly signaled by the Java environment due to programming mistakes or resource limitations (as the ones found in our study).
Identifying the Concerns Related to Root Exceptions
To get a broader view of the root exceptions of stack traces, we performed a manual inspection in order to identify the underlying concerns related to the most frequently reported root exceptions. Besides the exceptions related to programming mistakes mentioned before, we also looked for exceptions related to concerns that are known as sources of faults in mobile development: concurrency (Amalfitano et al. 2012) backward compatibility (McDonnell et al. 2013), security (Enck et al. 2011; Wasserman 2010) and resource management (IO, Memory, Battery) (Zhang and Elbaum 2012). Since it is infeasible to inspect the code responsible for throwing every exception reported in this study, the concern identification for each exception was based on intended meaning of the particular exception type, as defined in its Javadoc documentation and in the Java specification. For example: (i) an instance of ArrayOutOfBoundException refers to a programming mistake according to its Javadoc; and (ii) the Java specification lists all exceptions related to backward compatibility,5 such as InstantiationError, VerifyError, and IllegalAccessError.
Identifying the concerns related to root exceptions
% Occurrences on stacks
Programming logic (java.lang and util)
Resources (IO, Memory, Battery)
General (Error, Exception, Runtime)
RQ 1.2 Can the exception types reveal bug hazards?
Types and origins of root exceptions
Inspecting Exception Interfaces
Absence of exception interfaces in methods
Missing Checked Exceptions in Exception Interfaces
RQ 1.3 Can the exception wrappings reveal bug hazards?
Wrappings comprising different exception types
Runtime Exception Wrapping a Checked Exception
Runtime Exception Wrapping an Error
Examples of Cross-type wrappings
Runtime Exception wrapping an Error
java.lang.RuntimeException - java.lang.OutOfMemoryError
java.lang.RuntimeException - java.lang.StackOverflowError
Checked Exception wrapping an Error
java.lang.reflect.InvocationTargetException - java.lang.OutOfMemoryError
java.lang.Exception - java.lang.OutOfMemoryError
Error wrapping a Checked Exception
java.lang.NoClassDefFoundError - java.lang.ClassNotFoundException
java.lang.AssertionError - javax.crypto.ShortBufferException
Error wrapping a Runtime Exception
java.lang.ExceptionInInitializerError - java.lang.NullPointerException
java.lang.ExceptionInInitializerError - java.lang.IllegalArgumentException
Checked Exception Wrapping an Error
Most of these wrappings were also caused by the reflection library used by applications’ methods. The methods responsible for the wrappings were also native methods written in C. Table 8 illustrates some of these wrappings — some of them are masking an OutOfMemoryError into a checked exception. On the one hand, by confronting the API user with a checked exception, the API designer is forcing the client to handle the exceptional condition. On the other hand, according to the Java specification Errors are not supposed to be caught. In this case, even if the exception is caught by a handler, the problem that trigged the Error remains: there is not enough memory to execute the app. This kind of wrapping is even more dangerous than the previous one and may lead to the “exception confusion” described next.
Error Wrapping Runtime and Checked Exceptions
Table 8 illustrates examples of instances of Error wrapping instances of RuntimeException. Although such a wrapping mixes different exception types, since there is no obligation associated with handling runtime exceptions, it does not violate the aforementioned best practices.
On the other hand, the inspection also revealed instances of Error wrapping checked exceptions. Such wrappings were mostly performed by Java static initializers. If any exception is thrown in the context of a static initializer (i.e., static block) it is converted into an ExceptionInitializerError at the point where the class is first used. Table 8 also illustrates examples of such wrappings. Although such a wrapping may represent a design decision, it violates the best practices related to checked exceptions and errors as it mixes the intended handling behavior associated with both types.
4 The Developers’ Perspective
As previously mentioned, the goal of this work is to obtain a thorough understanding of common exception handling bug hazards that app developers face. In the first phase of this work, we conducted a mining study which identified common exception handling bug hazards in app development. In the second phase of this work, we set up an exploratory qualitative investigation and surveyed Android developers on how they perceive the bug hazards detected in the mining study. The scope of our study is GitHub—using our GHTorrent database (Gousios 2013), we aimed our survey at developers from Android projects available in GitHub for which issues were mined during the first phase of this study. Next sections detail this exploratory qualitative investigation.
4.1 Research Questions
The overall research question guiding our exploratory survey is the following: RQ 2: How do developers deal with the exception handling code in Android apps and what are the developers’ perspectives about the main bug hazards found during the mining study? This general research question has been broken into a set of research questions that are answered by the exploratory survey.
When developing Java-based applications it is inevitable to deal with exceptions. Hence, our first question explores how developers deal with the exception handling code in Android development:
- RQ 2.1:
How do developers deal with exception handling code while developing Android apps?
To make the analysis easier, we further refine this question into sub-questions, as follows. To investigate whether the development of exception handling code is a daily concern for developers or it is something that developers rarely face and for that reason do not care much, the following subquestions are added: How often do developers handle exceptions?How often do developers throw exceptions? And, to evaluate developers knowledge concerning EH best practices, we also investigated: Do developers know about Java EH best practices and/or Android specific EH best practices?
The subsequent research questions focus on the developers’ perspectives about the main bug hazards found during the mining study. We address these questions by presenting a set of code snippets in which the bug hazards are present, asking developers what they would do when faced with such scenarios. The questions related to bug hazards are as follows:
- RQ 2.2:
How do NullPointerExceptions impact the development of robust Android apps?
- RQ 2.3:
How do cross-type wrappings impact the development of robust Android apps?
- RQ 2.4:
Are developers aware of the robustness threats caused by JNI undocumented checked exceptions?
Our last research question addresses whether the exception handling code helps with the development of robust Android applications, and what developers usually do to prevent apps from crashing. The motivation behind this question was to discover common practices for dealing with uncaught exceptions and to assess developers perspective about the role of the exception handling mechanism in the development of robust apps. Hence, the last question is as follows:
- RQ 2.5:
How does the exception handling code affect the development of robust apps and how do developers prevent apps from crashing?
Given the exploratory nature of our research, we used methods from Grounded Theory (Charmaz 2006) to answer some of our research questions. Since our aim is to learn from a large number of developers, we use surveys which are known to scale well. The survey is split into two logical sections; the motivations behind each logical section are: (i) questions aiming at learning from developers about the usage of the exception handling code in app development; and (ii) questions focusing on getting developers’ perceptions about the bug hazards detected in the first phase of this study.
The survey comprises multiple choice or Likert-scale questions and open-ended questions. The multiple choice questions are intermixed with open-ended questions to further elicit the developer’s opinions. Moreover, the survey also contains Likert-scale questions to force participants to make a choice. Overall, the survey includes 13 open-ended questions, 5 Likert-scale questions and 10 multiple choice questions. The respondents could complete the survey in about 15 minutes.
We used grounded theory coding to iterate through the open-ended survey responses. The grounded theory coding used in this study consists of two phases: (1) initial coding entails a close reading of the data and (2) later we used focused coding (Charmaz 2006) to pinpoint and develop the most salient themes (Charmaz 2006) in the analyzed data. Answers to different questions in the survey were coded separately. For all open-ended questions in the survey, coding was done by two researchers until saturation was reached. Disagreements were discussed and resolved as part of the coding, thus, we are unable to report level of agreement. In all cases, the first author was one of the two coders, and one of the other authors coded the data as well in close collaboration with the first author until saturation was reached. The first author then went through the remaining responses to assign codes to responses.
RQ 2.1: How do developers deal with the exception handling in Android development?
This first research question explores the ways in which developers deal with the exception handling code, either throwing or handling exceptions, while developing an Android app. Developers were asked about the frequency at which they develop exception handling code and whether or not they adopt best practices while developing.
Top mentioned Java EH best practices – 40 non-empty responses
Top Java EH Best Practices
Use specific handlers / don’t catch generic exceptions
Don’t swallow Exceptions
Don’t throw Runtime / Favor Checked exceptions
Do not use exception for normal flow control
Free Resources in finally-blocks
Crash report tools
Don’t catch Errors
RQ 2.2: How do the NullPointerExceptions impact the development of robust Android apps?
Top mentioned Android-specific EH best practices – 14 non-empty responses
Top Android EH Best Practices
Same as Java
Crash report tools
Add a global exception handler (UncaughtExceptionHandler)
Use checked exceptions
Use appropriate exception messages
Developers reported some reasons for the high prevalence of crashes caused by NullPointerExceptions. The activity and fragment lifecycle was one of them – “Android destroys and recreates itself all of the time (especially during screen rotation) and if you do not handle that it will crash on you every time. with the complexity of an activity with a fragment that has fragments and each of those fragments has custom objects and variables that need to be either retained (so saved and put back) or recreated such as views it can get complex if you don’t have an understanding of how the android life cycle works.” [D43]. Another developer also mentioned that: “They [NullPointerExceptions] can happen pretty much anywhere. The Android Fragment system comes to mind in this case. Often, it is possible to find yourself in a state where getActivity() is null within the Fragment during certain points in the life cycle, and that is something I have to plan for. This might have been avoidable under a different structure.”[D56].
Top reasons why NullPointerExceptions are more frequent in Android apps according to 19 respondents
Activity/fragment life cycle
API / Backward compatibility
No layers to catch runtimeexceptions
Almost half of the developers mentioned, however, that NullPointerExceptions are common in Android development since they are common in Java standard development as well. One responded even mentioned: “Honestly, there are a lot of very unskilled Java programmers out there writing Android apps. When they encounter NPE, they tend to null-check that variable, which just puts a bandage on the problem and causes other failures (usually also NPE’s) later on in the application’s lifecycle. This is nothing specific to Android, it’s just how Java works”[D42].
Top ways for preventing NullPointerException in Android apps – 61 non-empty responses
Top Ways of Preventing NullPointerExceptions
Investigate/fix the cause
Catch null-pointer (mistake)
Initialize/use default variable
New control-flow for null
Avoid using nulls / avoid to use methods can throw null
Some of the best practices are related to the different ways an exception should be handled according to whether it is a checked or an unchecked exception. When the designer of an API specifies that a method throws a checked exception, it is telling the caller of the API that such an exception should be handled.
RQ 2.3: How do Cross-Type Wrappings impact the development of robust Android apps?
To assess the developers’ perspective on whether checked and unchecked exceptions should be handled differently inside the app, the developers were presented with two pieces of code: one in which a RuntimeException was caught and another one where an IOException was caught; both in the context of Activity life-cycle methods. Then the developers were asked whether the way to handle a runtime exception should be different from the way to handle a checked exception.
Top ways of dealing with a checked exception signaled in the context of an Activity method – 40 non-empty responses
Top Ways of Handling Checked Exceptions
Add a try-catch block
Present error message
Involve the user in solving
Investigate the cause
Same way as runtime
Add a throws declaration / pass upstream
Top ways of handling runtime exceptions – 63 non-empty responses
Top Ways of Handling Runtime Exceptions
Add a try-catch block
Present error message
Log the exception
Throw a checked exception
Let it crash / crash fast
Swallow the exception
Add a throws declaration
However, what developers mentioned as handling actions (what is performed inside the catch clause to deal with the exception) differs in both cases. For runtime exceptions most of the handling actions were dedicated to present more specific error messages and prevent the app from abruptly crashing. “[...] They signal exceptional behavior that may not be recoverable, so they offer a useful way to log and gracefully crash” [D19]. “Giving the user meaningful feed back is important, so the more specific you can be about what went wrong, the better.”[D66]. Some developers also mentioned to use a toast to support this task.9 Few developers also mentioned that the runtime exception should flow upstream to the framework so it could crash the app.
On the other hand, some handling actions mentioned for checked exceptions were: (i) retrying the same operation; (ii) involving the user in finding a solution to the exception condition such as opening a pop-up and asking the user to define a new place to save the file (the example presented to them involved a IOException being signaled); and also (iii) presenting an error message to the user.
We can observe that the handling actions associated with checked exceptions focused more on crash prevention than the ones related to runtime exceptions. “Its very important in android that we gracefully handle an exception. Try not to crash an application.”. It was also interesting to observe that the developers suggested involving the user and solving the exceptional condition represented by a checked exception – “We should try alternative saving ways and if it fails we should prompt the user for a new location and always keep the user informed”[D11]. “You still need to catch an IOException to prevent a crash. However, because it is related to saving a file, you may need to reset any data within that catch statement and either alert the user that it has failed or perform a limited retry (in the case of an HTTP upload or something prone to server-side failure).”[D5].
Although there is a long lasting debate about the pros and cons of checked and unchecked exceptions, the survey revealed that many Android developers considered checked exceptions as a way of using exceptions that can prevent uncaught exception crashes – since in order for a checked exception to flow upstream and crash the app the developers need to explicitly do so.
Top reasons why cross-type wrapping may affect app robustness – 49 non-empty responses
Top Reasons Why Cross-Type Wrapping Affect Robustness
Impairs proper handling (loses exception information)
Uncaught will crash the app
App will crash anyway
Ahould catch / handle properly (do local recovery)
Treat all exceptions as critical
Activity methods cannot throw exceptions
RQ 2.4: Are developers aware of the robustness threats caused by JNI undocumented checked exceptions?
The respondents were presented with a piece of code in which a non-declared checked exception has appeared, and they were asked if they knew any reason for this to happen. Only 3 respondents out of 71 (4 % of respondents) were aware of the fact that there are ways a method can throw a non-declared checked exception, such as JNI/native code, reflection, and directly changing the bytecode/dalvik code, which bypass the compile-time checking. One of them said: “I am guessing this is because the exception is thrown from native code (i.e., C++ code in the JVM) where Java correct-ness semantics rules do not always apply.” [D42].
RQ 2.5: How does the exception handling code affect the development of robust apps and how do developers prevent apps from crashing?
Developers were also asked whether the exception handling code helped with the development of robust Android applications – see Fig. 6. Although most of the developers answered positively (78.9 %), when asked to provide one or more reasons for their answers, most of the respondents also provided reasons why the exception handling code can sometimes impair the robustness.
Top 5 reasons why exception handling helps or impairs the development of robust apps
For Positively Affecting Robustness
Improves error diagnosis
Anticipated erroneous situation help writing more robust code
Exceptions only for unrecoverable behavior
Checked exceptions force handling
Useful to gracefully crash
For Negatively Affecting Robustness
Crashes the app if not handled
Makes debugging harder
Unchecked/runtime exceptions may crash the app
NullPointerExceptions can happen anywhere/are tricky to avoid
Although most of the respondents answered that exception handling code helped with the development of robust Android applications, almost every respondent pointed out a drawback associated with the exception handling code, saying that if it is not used with care the exception handling code may lead to app crashes. “Any uncatched exception will crash the app. There are many unknown exception thrown by the system, that are only happen once in a lifetime like IllegalStateException: eglMakeCurrent failed EGL_BAD_CONTEXT”[D32]. “The lifecycle of activitiesfragments etc make it harder to work out what order things are called in and so exceptions can occur because you didn’t realise that another method isn’t called”[D70].
Finally developers were asked how they prevent their apps from crashing. Most of the respondents answered that they prevent apps from crashing by handling exceptions/errors (27 %). They mentioned strategies such as: handling exceptions in all entry points/all over; using catch-all clauses; and adding try-catch blocks around risky methods. Different from conventional Java programs, an Android app is composed of several entry points, each window (i.e. Activity) is a potential entry point in which exceptions can arise leading to an app crash. This explains the importance of handling exceptions in all entry points, and using general catch clauses which sometimes cannot prevent crashes but allow the developer to present a detailed error message to the user before crashing. “You can wrap the main function in java in try/catch in android you can’t...”[D12]. “Though not particular to Android, most Android apps have several different entry points: Activities, Services, etc, can be started by different services; same thing with events. It may be hard to ensure that all this cases are properly handled” [D46].
Top ways of preventing crashes – 71 non-empty responses
Top Ways of Crash Prevention
- null check annotations (@Null and @NotNull)
Crash report / crash report tools
“Everybody hates thinking about exceptions, because they are not supposed to happen” (Brian Foote)12
This section discusses the lessons learned from the mining and the survey studies. The mining study revealed a set of bug hazards – such as (i) the cross-type wrappings; (ii) the abundance of null pointer problems; and (iii) the undocumented runtime exceptions signaled by third-party code, which were confirmed by developers during the survey. Here, we discuss the identified bug hazards, we present how the developers perceived them and point out at what developers can do in order to deal with them.
5.1 The Exception Handling Confusion Problem
When (mis)applied, exception wrapping can make the exception-related code more complex and lead to what we call the exception handling confusion problem. This problem can lead a program to an unpredictable state in the presence of exceptions, as illustrated by the scenario in which a checked exception wraps an OutOfMemoryError. Currently there is no way of enforcing Java exception type conventions during program development. Hence, further investigation is needed on finding ways to help developers in dealing with this problem, either preventing odd wrappings or enabling the developer to better deal with them. Furthermore, only some of the Android developers surveyed in this study were aware of the robustness threats caused by cross-type wrappings as the one cited above. This calls for empirical studies on the actual usefulness of Java’s hybrid exception model.
5.2 On the Null Pointer Problem
The null reference was first introduced by Tony Hoare in ALGOL W, which after some years he called his “one-billion-dollar mistake” (Null references:the billion dollar mistake, abstract of talk at QCon London ??). In this study, the null references were, in fact, responsible for several reported issues – providing further evidence to Hoare’s statement. Moreover many of the survey respondents recognized that NullPointerExceptions were one of the main causes of application crashes since they can happen almost anywhere in the application code, and to make things worse the life-cycle of Android apps in which objects are constantly recreated (i.e., Activity and Fragment classes) favors this kind of exception to happen added to the fact that many Android framework methods return null without making this return type explicit in the documentation. Such observations emphasizes the need for solutions to avoid NullPointerExceptions, such as: (i) lightweight intra-method null pointer analysis as supported by Java 8 @Nullable annotations;13 (ii) inter-method null pointer analysis tools such as the one proposed by Nanda and Sinha (2009); or (iii) language designs which avoid null pointers, such as Wadler (1995) (as used in functional languages for values that may not be available or computations that may fail), to improve the robustness of Java programs. Some of the Android developers mentioned that @Nullable annotations could be helpful to deal with NullPointerExceptions and consequently prevent app crashes caused by them.
5.3 Preventing Uncaught Exceptions
In this study we could observe undocumented runtime exceptions thrown by third party code, and even undocumented checked exceptions thrown by a JNI interface. Such undocumented exceptions make it difficult, and most of the times infeasible, for the client code to protect against “unforeseen” situations that may happen while calling library code. In the survey-based study Android developers were asked about ways to prevent application crashes which are mainly caused by uncaught exceptions. Many of them emphasized the importance of handling exceptions to prevent crashes but also mentioned how difficult it is to handle every specific exception that can happen. Several developers follow a more reactive behavior against crashes: they advocate the use of crash report tools, and once the crash happens for the first time, it is reported and the application can be changed to better cope with the exceptions that caused the crashes, consequently preventing future similar crashes.
One may think that the solution for the uncaught exceptions may be to define a general handler, which is responsible for handling any exception that is not adequately handled inside the applications. Although this solution may prevent the system from abruptly crashing, as mentioned by some of the surveyed developers such a general handler will not have enough contextual information to adequately handle the exceptions, beyond storing a message in a log file and restarting the application. Such a handler cannot replace a carefully designed exception handling policy (Robillard and Murphy 2000), which requires third-party documentation on the exceptions that may the thrown by APIs used. Since documenting runtime exceptions is a tedious and error prone task, this calls for tool support to automate the extraction of runtime exceptions from library code. Initial steps in this direction have been proposed by Van Dooren and Steegmans (2005).
5.4 Mining Study – Threats to Validity
We used a heuristics-based parser to mine exceptions from issues. Our parsing strategy was conservative by default; for example, we only considered exception names using a fully qualified class name as valid exception identifiers, while, in many cases, developers use the exception name in the issue description. Conservative parsing may minimize false positives, which was our initial target, but also tends to increase false negatives, which means that some cases may have not been identified as exceptions or stack traces. Our limited manual inspection did not reveal such cases. Moreover, in this study we manually mapped the concerns related to exceptions. To ensure the quality of the analysis, we calculated the interrater agreement after three independent raters classified a randomly selected sample (of 25 exception types from the total of 100); the interrater agreement was high (96 %).
The process for identifying the type of exceptions reported in stack traces may not be completely accurate. Specifically, we performed the selection of the version of the exception source code or bytecode to analyze, as follows: (i) for all exceptions signaled by the Android Platform and the Java Environment the analysis was based on Version 4.4 of Android platform (API level 19); (ii) for the exceptions signaled by applications, the analysis considered the last version of the application source-code available on Github/Google code; and (iii) for all exceptions signaled by third-party libraries the type analysis considered the latest bytecode version available on the app repository. When the exception could not be found automatically or manually (based on the chosen version), we classified the exception as “Undefined”. Only 31 exceptions remained undefined, which occurred in 60 different exception stack traces out of 6,005 mined stack traces. Hence, the exception type could not be accurately identified in 1 % of the mined exception flows. There should be situations where the exception type (i.e., checked or runtime) changes from one version to another. Although we believe that it will not happen very often, we did not investigate this issue.
Our work uses the GHTorrent dataset, which although comprehensive and extensive is not an exact replica of GitHub. However, the result of this study does not depend on the analysis of a complete GitHub dataset. Instead, the goal of our study was to pinpoint bug hazards in the exception-related code based on exception stack trace mining of a subset of projects. We limited our analysis to a subset of existing open-source Android projects. We are aware that the exception stack traces reported for commercial apps can be different from the ones found in this study, and that this subset is a small percentage of existing apps. Such threats are similar to the ones of other empirical studies which also use free or open-source Android apps (Linares-Vásquez et al. 2013; McDonnell et al. 2013; Ruiz et al. 2012). Moreover, several exception stack traces that support the findings of this study referred to exceptions coming from methods defined in the Android Application Framework and third-party libraries. Additionally, the bug hazards observed in this study are due to characteristics of the Java exception model, which can impose challenges to the robustness of not only Android apps but also to other systems based on the same exception model.
Another threat relates to the fact that parts of our analysis are based on the availability of stack traces in issues reported on GitHub and Google Code projects. In using these datasets, we make an underlying assumption: the stack traces reported in issues are representative of valid crash information of the applications. One way to mitigate this threat would be to access the full set of crash data per application. Although some services exist to collect crash data from mobile applications (e.g., ACRA,14 Google analytics,15 Bugsense, Bugsnag16), they do not provide open access to the crash reports of their client applications. In our study, we mitigated this threat by manually inspecting the source code associated with a subset of the reported exception stack traces. This subset comprises the stack traces related to the main findings of the study (e.g., “undocumented runtime and checked exceptions”, and “cross-type wrappings”).
5.5 Limitations of the Survey-based Study
Due to the exploratory nature of the second phase of this work whose goal was to identify the developers’ perspectives concerning the exception handling bug hazards found during the repository mining phase, we chose Grounded Theory techniques. The results of our survey-based study may not apply to every Android developer, since other populations might add new insights. The population we collected data from was comprised of Android developers of the GitHub Android projects whose issues were mined in the first phase of this work, and who had time and motivation to answer the survey questions. Although the themes and findings that emerged in our study cannot be generalized, they give a first view of developers’ perspectives about EH bug hazards found. Hence, we believe that the survey-based study has contributed with valuable insights about how Android developers used the exception handling code and what their perspectives regarding exception handling bug hazards are.
Although the survey was applied to developers who had contributed to at least one of the GitHub apps, analyzed in the mining study, the survey was not customized to each developer. In other words, it did not contain questions focusing on specific bug hazards identified in their apps. That customization of the survey (for each specific developer/app) could give more insights about the exception handling usage and bug hazards, and could even have improved the response rate for the survey. However, we did not followed this approach because the high number of projects analyzed (482 Android apps) and their contributors (1,824 developers) would increase the complexity and time needed (i) to prepare the survey and specially (ii) to analyze the survey responses - since themes could emerge from specific contexts. This approach can be very useful in a guided interview involving some of the respondents to refine the findings of this exploratory survey in a future work.
5.6 Replication Package
All the data used in the mining study and in the survey-based study is publicly available at the ExceptionMiner tool website hosted on GitHub.17 Specifically we provide: (i) all issues related to Android projects found on GitHub and Google Code used in this study; (ii) all stack traces extracted from issues; (iii) the results of manual inspection steps; (iv) the ExceptionMiner tool we developed to support stack trace extraction and distilling; (v) the survey questions; and (iv) the survey responses and summary.
6 Related Work
In this section, we present work that is related to the present paper, divided into four categories as detailed next.
Analysis and Use of Stack Trace Information
Several papers have investigated the use of stack trace information to support: bug classification and clustering (Wang et al. 2013; Kim et al. 2011; Dhaliwal et al. 2011), fault prediction models (Kim et al. 2013), automated bug fixing tools (Sinha et al. 2009) and also the analysis of Android APIs (Kechagia and Spinellis 2014). Kim et al. (2011) use an aggregated form of multiple stack traces available in crash reports to detect duplicate crash reports and to predict if a given crash will be fixed. Dhaliwal et al. (2011) proposed a crash grouping approach that can reduce bug fixing time by approximately 5 %. Wang et al. (2013) propose an approach to identify correlated crash types and describe a fault localization method to locate and rank files related to the bug described in a stack trace. Schröter et al. (2010) conducted an empirical study on the usefulness of stack traces for bug fixing and showed that developers fixed the bugs faster when failing stack traces were included in bug issues. In a similar study, Bettenburg et al. (2008a) identify stack traces as the second most relevant feature of good bug reports. Sinha et al. (2009) proposed an approach that uses stack traces to guide a dataflow analysis for locating and repairing faults that are caused by the implicitly signaled exceptions. Kim et al. (2013) proposed an approach to predict the crash-proneness of methods based information extracted from stack traces and methods’ bytecode operations. They observed that most of the stack traces were related to NullPointerException and other implicitly thrown exceptions had the higher prevalence in the analyzed set of stacks. Kechagia and Spinellis (2014) examined the stack traces embedded in crash reports sent by 1,800 Android apps to a crash report management service (i.e., BugSense). They found that 19 % of such stack traces were caused by unchecked and undocumented exceptions thrown by methods defined in the Android API (level 15). Our work differs from Kechagia and Spinellis since it is based on stack traces mined from issues reported by open source developers on GitHub and Google Code. Moreover, our study mapped the origin of each exception (i.e., libraries, the Android platform or the application itself) and investigated the adoption of best practices based on the analysis of stack trace information. Our work also identified the type of each exception mined from issues (classifying them as Error, Runtime or Checked) based on the source code analysis of the exception hierarchy and analyzed the exception wrappings that can happen during the exception propagation. Such analysis revealed intriguing bug hazards such as the cross-type exception wrappings not discussed in previous works.
Extracting Stack Traces from Natural Language Artifacts
Apart from issues and bug reports, stack traces can be embedded in other forms of communication between developers, such as discussion logs and emails. Few tools have been proposed to mine stack traces embedded on such resources. Bettenburg et al. (2008b) is based on a set of regular expressions that extract a set of frames related to a stack trace. The main limitation of this solution is that it is not able to extract stack traces embedded in verbose log files (i.e., in which we can find log text mixed with exception frames). Bacchelli et al. (2012) propose a solution to recognize stack trace frames from development emails and relate them to code artifacts (i.e. classes) mentioned in the stack trace. In addition to those tools, ExceptionMiner is able to both extract stack traces from natural language artifacts and to classify them into a set of predefined categories.
Empirical Studies on Exception Handling Defects
Cabral and Marques (2007) analyzed the source code of 32 open-source systems, both for Java and .NET. They observed that the actions inside handlers were very simple (e.g., logging and presenting a message to the user). Coelho et al. (2008) performed an empirical study considering the fault-proneness of aspect-oriented implementations for handling exceptions. Two releases of both Java and AspectJ implementations were assessed as part of that study. Based on the use of an exception flow analysis tool, the study revealed that the AOP refactoring increased the number of uncaught exceptions, degrading the robustness of the AO version of every analyzed system. The main limitation of approaches based on static analysis approaches are the number of false positives they can generate, and the problems faced when dealing with reflection libraries and dynamic class loading. Zhang and Elbaum (2012) were the first to perform an empirical investigation of issues, related to exception-related bugs, in Android projects. They perform a small scale study in which they manually inspected the issues of 5 Android applications. They observed that 29 % had to do with poor exceptional handling code, and this empirical study was used to motivate the development of a tool aiming at amplifying existing tests to validate exception handling code associated with external resources. This work inspired ours, which automatically mined the exception stack traces embedded in issues reported in 639 open source Android projects. The goal of our study was to identify common bug hazards in the exception related code that can lead to failures such as uncaught exceptions.
Empirical Studies Using Android Apps
Ruiz et al. (2012) investigated the degree of reuse across applications in the Android Market; the study showed that almost 23 % of the classes inherited from a base class in the Android API, and that 217 mobile apps were reused completely by another mobile app. Pathak et al. (2011) analyzed bug reports and developers’ discussions of the Android platform and found that approximately 20 % of energy-related bugs in Android occurred after an OS update. McDonnell et al. (2013) conducted a case study of the co-evolution behavior of the Android API and 10 dependent applications using the version history data found in GitHub. The study found that approximately 25 % of all methods in the client code used the Android API, and that the methods reusing fast-evolving APIs were more defect prone than others. Linares-Vásquez et al. (2013) analyzed approximately 7K free Android apps and observed that the least successful apps used Android APIs that were on average 300 % more change-prone than the APIs used by the most successful apps. Our work differs from the others as it aims at distilling stack trace information from bug reports and combining such information with bytecode analysis, source code analysis and manual inspections to identify bug hazards in the exception handling code of Android apps.
Exploratory Survey Studies
Exploratory surveys have been used in the software engineering context to discover the user perspective regarding a broad range of topics such as: assessing the developers’ perceptions on productivity (Meyer et al. 2014); how GitHub developers use pull-requests (Gousios et al. 2015); how the testing culture of open-source projects can be characterized (Pham et al. 2013); and even how developers use Twitter (Singer et al. 2014). This kind of study is important as a way of better understanding the developers’ behavior and hence providing recommendations and tools to help with specific development tasks.
Exploratory surveys have also targeted Android developers (Kochhar et al. 2015; Joorabchi et al. 2013; Bavota et al. 2015; Linares-Vásquez et al. 2015). Kochhar et al. (2015) conducted a survey-based study to discover the commonly used tools for mobile app testing as well as the problems faced by developers while testing the apps. The survey was sent to 3,905 emails of Android developers and received 83 responses (response rate of 2.13 %). (Joorabchi et al. 2013) performed a survey-based study whose goal was to better understand the main challenges developers face when building apps for different mobile devices. They interviewed 12 senior mobile developers and performed a semi-structured survey, with 188 respondents from the mobile development community, since the survey was shared via e-mail groups and social media websites response rate of the survey could not be calculated. Linares-Vásquez et al. (2015) surveyed 485 open source Android app and library developers (for projects hosted on GitHub) to understand developers’ practices for detecting and fixing performance bottlenecks in mobile apps. This work emailed the survey to 24,340 developers and received 628 responses - the response rate was 2.6 %. Bavota et al. (2015) investigated the impact of API change-proness and fault-proneness on the user ratings of Android apps. They surveyed developers to assess their perspective on whether such problems could be the cause for unfavorable user ratings. The response rate was 4 %. A common characteristic among such works and our work is the low response rate of the surveys (between 4 % and 2 %). None of these surveys, however, assessed how developers deal with exceptions in Android nor the developers’ perspective regarding a set of exception handling bug hazards. In our study, we conduced the first exploratory survey focusing on exception handling issues in Android development.
The goal of this paper is two-fold: (i) to investigate to what extent stack trace information can reveal bug hazards related to exception handling code that may lead to a decrease in application robustness; and (ii) to assess the developers’ perspective concerning the detected exception handling bug hazards.
To realize this goal, we mined the stack traces embedded in all issues defined in 482 Android projects hosted on GitHub and 157 projects hosted on Google Code – overall considering 6,005 exception stack traces. We subsequently surveyed developers associated with a selection of the mined GitHub projects.
Our first key contribution is a novel approach and toolset (ExceptionMiner) for analyzing Java exception stack traces as occurring in GitHub and Google Code issues.
Half of the system crashes are due to errors in programming logic, with null pointer exceptions being most prominent;
Documentation for explicitly thrown RuntimeExceptions is almost never provided;
Extensive use of wrapping leads to hard-to-understand chains violating Java’s exception handling principles.
Our third contribution is a qualitative study to assess the developers’ perspective concerning the exception handling bug hazards in Android development. This study corroborates the findings of our stack trace analysis, most notably the prevalence of null pointer exceptions and the reliability implications of (in particular cross-type) wrappings. Furthermore, we found that few developers are aware of the undocumented checked exceptions signaled by native C code of the Android platform.
In conclusion, our findings shed light on common problems and bug hazards in Java exception handling code, and call for tool support to help developers understand their own and third party exception handling and wrapping logic.
K9Mail moved to GitHub but as a way of not loosing the project history it advises their users to report bugs in the Google Code issue tracker: https://github.com/k9mail/k-9/wiki/LoggingErrors.
2 Google Code used to provide a Web service to its repositories, but this was deactivated in June 2013 in what Google called a “clean-up action”.
3 In several exception stack traces, the exception frames were preceded by logging information e.g., 03-01 15:55:01.609 (7924): at android.app.ActivityThread.access$600(Activity Thread.java:127) which could not be detected by existing tools.
6 The filtering was performed in two steps: firstly, we analyzed the bytecode of the JVM and identified all runtime exceptions defined by it (e.g., java.lang.NullPointerExceptoin,java.lang.ArrayIndexOutOfBounds), then if the libraries/framework method was signaling one of such exceptions it was filtered out from the analysis.
9 A toast is an Android component that provides simple feedback about an operation in a small popup – http://developer.android.com/guide/topics/ui/notifiers/toasts.html.
This cross-type wrapping was found in several applications during the mining study as well as in some classes of the Android framework.
Brian Foote shared his opinion in a conversation with James Noble – quoted in the paper: hillside.net/plop/2008/papers/ACMVersions/coelho.pdf.
Already supported by tools such as Eclipse, IntelliJ, Android Studio 0.5.5 (release Apr. 2014) to detect potential null pointer dereferences at compile time.
This work is partially supported by the National Institute of Science and Technology for Software Engineering (INES), CNPq and FACEPE, grants 573964/2008-4, 552645/2011-7, and APQ-1037-1.03/08, CNPq Universal grant 484209/2013-2, and CAPES/PROAP.
- Amalfitano D, Fasolino AR, Tramontana P, De Carmine S, Memon AM (2012) Using gui ripping for automated testing of android applications. In: Proceedings of the 27th IEEE/ACM International conference on automated software engineering. ACM, pp 258–261Google Scholar
- Bacchelli A, Dal Sasso T, D’Ambros M, Lanza M (2012) Content classification of development emails. In: Proceedings of ICSE 2012, pp 375–385Google Scholar
- Bettenburg N, Just S, Schröter A, Weiss C, Premraj R, Zimmermann T (2008a) What makes a good bug report? In: Proceedings of FSE 2008, pp 308–318Google Scholar
- Bettenburg N, Premraj R, Zimmermann T, Kim S (2008b) Extracting structural information from bug reports. In: Proceedings of MSR 2008. ACM, pp 27–30Google Scholar
- Binder R (2000) Testing object-oriented systems: models, patterns, and tools. Addison-Wesley ProfessionalGoogle Scholar
- Bloch J (2008) Effective java. Pearson Education IndiaGoogle Scholar
- Brunet J, Guerrero D, Figueiredo J (2009) Design tests: an approach to programmatically check your code against design rules. In: Proceedings of new ideas and emerging research (NIER) track at the international conference on software engineering (ICSE). IEEE, pp 255–258Google Scholar
- Cabral B, Marques P (2007) Exception handling: a field study in Java and.Net. In: Proceedings of ECOOP 2007. Springer, pp 151–175Google Scholar
- Charmaz K (2006) Constructing grounded theory: a practical guide through qualitative research. SagePublications Ltd, LondonGoogle Scholar
- Jenkov Tutorials. Checked or Unchecked Exceptions? (2014) http://tutorials.jenkov.com/java-exception-handling/checked-or-unchecked-exceptions.html, online
- Coelho R, Rashid A, Garcia A, Ferrari F, Cacho N, Kulesza U, von Staa A, Lucena C (2008) Assessing the impact of aspects on exception flows: An exploratory study. In: Proceedings of European conference on object-oriented programming (ECOOP). Springer-Verlag, pp 207–234Google Scholar
- Dhaliwal T, Khomh F, Zou Y (2011) Classifying field crash reports for fixing bugs: A case study of mozilla firefox. In: Proceedings of international conference on software maintenance (ICSM 2011), pp 333–342Google Scholar
- Enck W, Octeau D, McDaniel P, Chaudhuri S (2011) A study of android application security. In: USENIX security symposium, vol 2, pp 2Google Scholar
- Fraser G, Arcuri A (2013) 1600 faults in 100 projects: automatically finding faults while achieving high coverage with evosuite. In: Empirical software engineering, pp 1–29Google Scholar
- Garcia A, Rubira C, et al. (2007) Extracting error handling to aspects: a cookbook. In: Proceedings international conference on software maintenance (ICSM). IEEE, pp 134–143Google Scholar
- Gosling J (2000) The Java language specification. Addison-Wesley ProfessionalGoogle Scholar
- Gousios G (2013) The GHTorrent dataset and tool suite. In: Proceedings of the international working conference on mining software repositories (MSR). IEEE, pp 233–236Google Scholar
- Gousios G, Zaidman A, Storey MA, Van Deursen A (2015) Work practices and challenges in pull-based development: the integrator’s perspective. Tech. repGoogle Scholar
- Joorabchi M E, Mesbah A, Kruchten P (2013) Real challenges in mobile app development. In: 2013 ACM/IEEE International symposium on empirical software engineering and measurement. IEEE, pp 15–24Google Scholar
- Kechagia M, Spinellis D (2014) Undocumented and unchecked: exceptions that spell trouble. In: Proceedings of the 11th working conference on mining software repositories. ACM, pp 312–315Google Scholar
- Kim S, Zimmermann T, Nagappan N (2011) Crash graphs: an aggregated view of multiple crashes to improve crash triage. In: Proceedings of the IEEE/IFIP International conference on dependable systems and networks (DSN). IEEE, pp 486–493Google Scholar
- Kim S, Zimmermann T, Premraj R, Bettenburg N, Shivaji S (2013) Predicting method crashes with bytecode operations. In: Proceedings of the 6th India software engineering conference, pp 3–12Google Scholar
- Ko AJ, DeLine R, Venolia G (2007) Information needs in collocated software development teams. In: Proceedings of the 29th international conference on software engineering. IEEE Computer Society, pp 344–353Google Scholar
- Kochhar PS, Thung F, Nagappan N, Zimmermann T, Lo D (2015) Understanding the test automation culture of app developers. In: 2015 IEEE 8th International conference on software testing, verification and validation (ICST). IEEE, pp 1–10Google Scholar
- Linares-Vásquez M, Bavota G, Bernal-Cárdenas C, Di Penta M, Oliveto R, Poshyvanyk D (2013) API change and fault proneness: a threat to the success of Android apps. In: Proceedings of FSE 2013. ACM, pp 477–487. doi:10.1145/2491411.2491428
- Linares-Vásquez M, Vendome C, Luo Q, Poshyvanyk D (2015) How developers detect and fix performance bottlenecks in android apps. In: 2015 IEEE International conference on software maintenance and evolution (ICSME). IEEE, pp 352–361Google Scholar
- Maji A K, Arshad F A, Bagchi S, Rellermeyer JS (2012) An empirical study of the robustness of inter-component communication in Android. In: Proceedings of the IEEE/IFIP international conference on dependable systems and networks (DSN). IEEE, pp 1–12Google Scholar
- Mandrioli D, Meyer B (1992) Advances in object-oriented software engineering. Prentice-Hall IncGoogle Scholar
- McDonnell T, Ray B, Kim M (2013) An empirical study of api stability and adoption in the android ecosystem. In: Proceedings international conference on software maintenance (ICSM), pp 70–79Google Scholar
- Meyer AN, Fritz T, Murphy GC, Zimmermann T (2014) Software developers’ perceptions of productivity. In: Proceedings of the 22nd ACM SIGSOFT international symposium on foundations of software engineering. ACM, pp 19–29Google Scholar
- Miller R, Tripathi A (1997) Issues with exception handling in object-oriented systems. In: Proceedings of ECOOP’97. Springer, pp 85–103Google Scholar
- Nanda MG, Sinha S (2009) Accurate interprocedural null-dereference analysis for java. In: 31st International conference on software engineering, 2009. ICSE 2009. IEEE, pp 133–143Google Scholar
- Null references:the billion dollar mistake, abstract of talk at QCon London (2009) https://qconlondon.com/london-2009/qconlondon.com/london-2009/presentation/Null%2BReferences_%2BThe%2BBillion%2BDollar%2BMistake.html, online
- Pathak A, Hu YC, Zhang M (2011) Bootstrapping energy debugging on smartphones: a first look at energy bugs in mobile devices. In: Proceedings of the 10th ACM workshop on hot topics in networks. ACM, New York, HotNets-X, pp 5:1–5:6. doi:10.1145/2070562.2070567
- Pham R, Singer L, Liskin O, Figueira Filho F, Schneider K (2013) Creating a shared understanding of testing culture on a social coding site. In: 2013 35th International conference on software engineering (ICSE). IEEE, pp 112–121Google Scholar
- Robillard MP, Murphy GC (2000) Designing robust Java programs with exceptions. In: Proceedings international conference on the foundations of software engineering (FSE). pp 2–10Google Scholar
- Ruiz I, Nagappan M, Adams B, Hassan A (2012) Understanding reuse in the Android market. In: Proceedings of the international conference on program comprehension (ICPC), pp 113–122. doi:10.1109/ICPC.2012.6240477
- Sacramento P, Cabral B, Marques P (2006) Unchecked exceptions: can the programmer be trusted to document exceptions. In: International conference on innovative views of.NET technologiesGoogle Scholar
- Schröter A, Bettenburg N, Premraj R (2010) Do stack traces help developers fix bugs? In: Proceedings working conference on mining software repositories (MSR). IEEE, pp 118–121Google Scholar
- Stackoverflow Q&A. Java: checked vs unchecked exception explanation. (2014) http://stackoverflow.com/questions/6115896/java-checked-vs-unchecked-exception-explanation, online
- Singer L, Figueira Filho F, Storey MA (2014) Software engineering at the speed of light: how developers stay current using twitter. In: Proceedings of the 36th international conference on software engineering. ACM, pp 211–221Google Scholar
- Sinha S, Shah H, Görg C, Jiang S, Kim M, Harrold MJ (2009) Fault localization and repair for Java runtime exceptions. In: Proceedings International symposium on software testing and analysis (ISSTA). ACM, pp 153–164Google Scholar
- The Java tutorial. Unchecked exceptions: The controversy. (2014) http://docs.oracle.com/javase/tutorial/essential/exceptions/runtime.html, online
- Wadler P (1995) Monads for functional programming. In: Advanced functional programming. Springer, pp 24–52Google Scholar
- Wang S, Khomh F, Zou Y (2013) Improving bug localization using correlations in crash reports. In: Proceedings working conference on mining software repositories (MSR 2013). ACM/IEEE, pp 247–256Google Scholar
- Wasserman AI (2010) Software engineering issues for mobile application development. In: Proceedings of the FSE/SDP workshop on future of software engineering research. ACM, pp 397–400Google Scholar
- Yuan D, Luo Y, Zhuang X, Rodrigues GR, Zhao X, Zhang Y, Jain P, Stumm M (2014) Simple testing can prevent most critical failures: An analysis of production failures in distributed data-intensive systems. In: 11th USENIX symposium on operating systems design and implementation, OSDI ’14. Broomfield, pp 249–265Google Scholar
- Zhang P, Elbaum S (2012) Amplifying tests to validate exception handling code. In: Proceedings international conference on software engineering (ICSE). IEEE Press, Piscataway, pp 595–605. http://dl.acm.org/citation.cfm?id=2337223.2337293