A field study of API learning obstacles

Abstract

Large APIs can be hard to learn, and this can lead to decreased programmer productivity. But what makes APIs hard to learn? We conducted a mixed approach, multi-phased study of the obstacles faced by Microsoft developers learning a wide variety of new APIs. The study involved a combination of surveys and in-person interviews, and collected the opinions and experiences of over 440 professional developers. We found that some of the most severe obstacles faced by developers learning new APIs pertained to the documentation and other learning resources. We report on the obstacles developers face when learning new APIs, with a special focus on obstacles related to API documentation. Our qualitative analysis elicited five important factors to consider when designing API documentation: documentation of intent; code examples; matching APIs with scenarios; penetrability of the API; and format and presentation. We analyzed how these factors can be interpreted to prioritize API documentation development efforts

Appendices

Appendix A: Exploratory Survey

Exact text of the survey described in Section 3.2 for the questions used in the research. This instrument also included questions whose answers were not included in the research reported in this article. These questions have been removed.

1.1 MSR API Learning Survey

The Human Interactions in Programming (HIP) group in Microsoft Research is conducting this survey to gather some initial data on Microsoft developers’ experiences with learning APIs. This data will be used in part to design tools and resources to help developers learn new APIs effectively. Your experiences and opinions would make a big difference in this research. Any personally-identifying information that you provide will be accessible only to the research team. For more information about this survey or the corresponding project, contact [the investigator].

This survey will take about 5 min to complete. Please refresh your memory about the API you learned the most recently. Published results will be anonymous. For your input to be useful, you must complete the survey by [deadline].

After taking the survey click “Submit” to save your changes. This survey is not anonymous

• 1. What is your primary job area?

• 2. How many total years of professional experience related to your job do you have?

• 3. How many of these years were at Microsoft?

This part of the survey will concern only your most recent experience with an API that you had to learn in your professional capacity. For this survey, consider that an API is a reusable set of program elements (classes, methods, functions, etc.) that are distributed and used together to provide higher-level functionality.

• 4. Name the last publicly-released API you learned:

• 5. What particular area of this API did you learn?

• 6. How many months ago did you start learning the API?

• 7. How many months ago did you last use the API?

• 8. How many hours in total would you estimate you’ve spent learning this API?

• 9. Were you also learning the programming language used to access the API? [Yes, No]

• 10. Were you familiar with the application domain of the API you learned? (For example, if you were learning WPF, were you already familiar with the creation of GUIs?) [Yes, No]

What obstacles made it difficult for you to learn the API? Obstacles can have to do with the API itself, with your background, with learning resources, etc. List the three most important obstacles, in order of importance (1 being the biggest obstacle). Please be more specific than the general categories mentioned here.

• 14. Obstacle 1:

• 15. Obstacle 2:

• 16. Obstacle 3:

• 17. Do you have any additional comments about learning APIs?

• 18. Would you be willing to participate in future research about this topic? [Yes, No]

Appendix B: Follow-up Survey

Exact text of the survey described in Section 3.4 for the questions used in the research. This instrument also included questions whose answers were not included in the research reported in this article. These questions have been removed.

1.1 API Learning Survey

The goal of this survey is to learn about your experience interacting with public APIs and to collect your opinions on how to improve API documentation. You can only complete this survey if you have had experience learning a public API that you remember enough to comment on. By “learning an API”, we mean using an API for the first time and engaging in activities that increase your knowledge of how this API works.

The survey should take less than 10 minutes to complete. As our appreciation for your time, participants who complete the survey will be entered in a draw for one $250 [on-line store] gift certificate.

The contest rules can be found at [internal URL]. After taking the survey click “Submit” to save your changes. This survey is not anonymous

• 1. How many years of professional experience do you have (decimals ok):

  API Learning Obstacles

This part of the survey applies only to the last public API you have learned.

• 2. What is the last public API you have learned? Pick the best match or choose “Other” (please specify below).

  [Azure; LINQ; Entity Framework; WPF; WCF; WF; CardSpace; Winforms; ASP.NET; ADO.NET; .NET Base Classes; Silverlight; XNA; Win32 (Admin); Win32 (Diagnostics); Win32 (Graphics); Win32 (Networking); Win32 (Security); Win32 (System Services); Win32 (UI); Other].

• 3. If you chose “Other”, please specify:

• 4. What programming language did you primarily use to access this API? [C/C $\boldsymbol{++}$ ; C#; Visual Basic; JScript, JavaScript, or other web language; Other]

• 5. If you chose “Other”, please specify:

• 6. What was the main reason for learning this API? [Your team started owning it; You needed to use it extensively as part of your work; You needed to use it to complete some specific tasks, but these were not the main part of your work; You learned the API to experiment with new technology related to your job; You learned the API for a hobby project, or for a side project not critical to your work; Other (please specify)]

• 7. Other:

For each type of obstacle described below, please rate how severe this type of obstacle was in your experience learning the API you mentioned above. [For questions 8–12 and 14, possible answers were as follows:]

[Blocker (Led to work being abandoned or a different APIs used, could not be realistically overcome); Very Severe (Led to significant delays and frustration, very difficult to overcome); Severe (Led to delays and frustration, difficult to overcome); Moderate (Led to delays and/or frustration, but could be overcome without excessive difficulty); Trivial (Was easy to overcome); This was not an obstacle at all]

• 8. When learning the API you mentioned above, how severe was this obstacle: Your background was not adapted to learning the new API. For example: you were not familiar with the programming language or application domain, your previous knowledge of a similar API (or a previous version of the API) made it confusing to learn the new API.

• 9. When learning the API you mentioned above, how severe was this obstacle: The way the API was structured or designed made it difficult to understand. For example: it was not clear how to instantiate an object, there were too many abstract classes, the names did not make sense.

• 10. When learning the API you mentioned above, how severe was this obstacle: The technical environment made it difficult to use the API. By technical environment, we mean any technical aspects not directly related to the design of the API itself. For example, the tools did not work well, the API required extensive infrastructure to test, you could not get the builds to work.

• 11. When learning the API you mentioned above, how severe was this obstacle: Specific member-level usage details were not documented. For example, description of parameters, error codes.

• 12. When learning the API you mentioned above, how severe was this obstacle: You did not find conceptual-level information explaining how to use the API. Consider “conceptual-level” information to mean any type of information you need to use the API correctly that is not typically associated with particular API members (classes/methods/functions). For example, description of required concepts, the API’s execution model, non-trivial code examples, usage patterns, best practices, mappings between scenarios and API members.

• 13. If you encountered a type of obstacle not included in the above list, please describe it here:

• 14. If you described an additional obstacle, how severe was this obstacle?

• 29. Do you have any additional ideas for improving high-level API documentation that could help developers learn how to use APIs more efficiently? If so please describe it here.

Appendix C: Quantitative Survey Results

• 2. What is the last public API you have learned? Pick the best match or choose “Other” (please specify below).

  API	Respondents
  Other	54
  LINQ	36
  WPF	34
  WCF	32
  .NET Base Classes	32
  Silverlight	24
  Win32 (System Services)	17
  Win32 (Networking)	16
  ASP.NET	14
  Win32 (Security)	14
  Winforms	10
  Azure	10
  Win32 (UI)	8
  Win32 (Graphics)	8
  XNA	8
  WF	5
  Entity Framework	4
  ADO.NET	4
  Win32 (Diagnostics)	3
  Win32 (Admin)	1

• 3. If you chose “Other”, please specify [Coded by categories]:

  API category	Respondents
  Operating System	16
  Development Tools	12
  Web	7
  Graphics/media	7
  Networking	5
  User Interface	3
  Unknown	3
  Business Applications	1

• 4. What programming language did you primarily use to access this API?

  Language	Respondents
  C#	229
  C/C++	97
  JScript, JavaScript, or other web language	2
  Other	5
  Visual Basic	1

• 6. What was the main reason for learning this API?

  API learning context	Respondents (%)
  You needed to use it extensively as part of your work	47
  You needed to use it to complete some specific tasks, but these were not the main part of your work	22
  You learned the API to experiment with new technology related to your job	17
  You learned the API for a hobby project, or for a side project not critical to your work	10
  Your team started owning it	3
  Other (please specify)	1

• 8–12. API Learning Obstacle Severity

  	Blocker	Very severe	Severe	Moderate	Trivial	Not an obstacle
  High-level documentation	1	9	27	37	17	10
  Low-level documentation	2	9	13	37	21	18
  Structure	0	4	11	43	27	15
  Technical environment	0	3	10	27	32	27
  Background	0	1	6	32	29	32