What should developers be aware of? An empirical study on the directives of API documentation


Application Programming Interfaces (API) are exposed to developers in order to reuse software libraries. API directives are natural-language statements in API documentation that make developers aware of constraints and guidelines related to the usage of an API. This paper presents the design and the results of an empirical study on the directives of API documentation of object-oriented libraries. Its main contribution is to propose and extensively discuss a taxonomy of 23 kinds of API directives.

This is a preview of subscription content, access via your institution.

Fig. 1
Fig. 2


  1. 1.

    Eclipse is an extensible development environment primarily targeted for Java.

  2. 2.

    See http://jgrapht.sourceforge.net.

  3. 3.

    See http://www.oracle.com/technetwork/java/javase/index.html.

  4. 4.

    See http://www.microsoft.com/net/.

  5. 5.

    See http://www.boost.org/.

  6. 6.

    See http://msdn.microsoft.com/en-us/library/d06h2x6e(v=VS.100).aspx.

  7. 7.

    This version of JDK is available at http://download.java.net/jdk6/6u21/promoted/b05/.

  8. 8.

    See http://wiki.eclipse.org/index.php/JFace.

  9. 9.

    $ cvs -d :pserver:anonymous@dev.eclipse.org/cvsroot/eclipse/ co -D “15 Oct 2009” org.eclipse.jface

  10. 10.

    See http://commons.apache.org/collections/.

  11. 11.

    To create and query the index, we used Lucene, see http://lucene.apache.net.

  12. 12.

    in Commons Collections’ CollatingIterator.addIterator(iterator)

  13. 13.

    in JDK’s RuntimePermission.RuntimePermission(name, actions)

  14. 14.

    in JFace’s IInformationControlExtension5.getInformationPresenterControlCreator()

  15. 15.

    in JDK’s Math.cos(a)

  16. 16.

    in JFace’s Dialog.getParent()

  17. 17.

    in JDK’s JobAttributes.getPageRanges()

  18. 18.

    in JDK’s RMIClassLoaderSpi.loadClass(codebase, name, defaultLoader)

  19. 19.

    in JFace’s PopupDialog.createDialogArea(parent)

  20. 20.

    in JDK’s Logger.setParent(parent)

  21. 21.

    in Commons Collections’ AbstractOrderedMapDecorator.AbstractOrderedMapDecorator()

  22. 22.

    in JFace’s ListEditor.getShell()

  23. 23.

    in JDK’s ActivationDesc.ActivationDesc(groupID, className, location, data, restart)

  24. 24.

    in JDK’s PolicySpi

  25. 25.

    in JDK’s LSParserFilter

  26. 26.

    in JDK’s Map.Entry.getKey()

  27. 27.

    in JDK’s DatabaseMetaData.getColumnPrivileges(catalog, schema, table, columnNamePattern)

  28. 28.

    in JFace’s ContentProposalAdapter.setAutoActivationCharacters(autoActivationCharacters)

  29. 29.

    Runtime.load(filename): “The filename argument must be a complete path name, (for example Runtime.getRuntime().load(“/home/avh/lib/libX11.so‘’);)”.

  30. 30.

    in JDK’s HttpCookie.HttpCookie(name, value)

  31. 31.

    in JDK’s ManagementPermission.ManagementPermission(name, actions). Although this may seem to be a bad design, we include it here since examples like this do appear in the APIs we studied.

  32. 32.

    in JDK’s Statement.execute(sql, autoGeneratedKeys)

  33. 33.

    in JDK’s ServerSocket.ServerSocket(port, backlog, bindAddr)

  34. 34.

    in JDK’s CallableStatement.registerOutParameter(parameterIndex, sqlType, scale)

  35. 35.

    in JDK’s DateFormat.format(obj, toAppendTo, fieldPosition)

  36. 36.

    in JDK’s CertPathValidator.validate(certPath, params)

  37. 37.

    in JDK’s MarshalledObject.MarshalledObject(obj)

  38. 38.

    in JDK’s KeyStore.setKeyEntry(alias, key, password, chain)

  39. 39.

    in JDK’s CallableStatement.setNClob(parameterName, reader, length)

  40. 40.

    in AlgorithmParameters.getInstance(algorithm)

  41. 41.

    in JDK’s File.renameTo(dest)

  42. 42.

    in JDK’s PortableRemoteObject.toStub(obj)

  43. 43.

    in JDK’s Runtime.addShutdownHook(hook)

  44. 44.

    in JDK’s ActivationGroup.inactiveObject(id)

  45. 45.

    in JFace’s LabelProvider

  46. 46.

    in Apache Commons Collections’ AbstractLinkedList.isEqualValue(value1, value2)

  47. 47.

    in JDK’s ThreadLocal.initialValue()

  48. 48.

    in JFace’s StyledCellLabelProvider

  49. 49.

    in Apache Commons Collections’ AbstractLinkedList

  50. 50.

    in JDK’s BigDecimal.setScale(newScale, roundingMode)

  51. 51.

    in JFace’s Dialog.createDialogArea(parent

  52. 52.

    in Apache Commons Collections’ Bag.remove(object)

  53. 53.

    in JFace’s StringFieldEditor.doFillIntoGrid(parent, numColumns)

  54. 54.

    in JDK’s Component.removeNotify()

  55. 55.

    in Apache Commons Collections’ AbstractLinkedList.init()

  56. 56.

    in JFace’s ErrorDialog.createDropDownList(parent)

  57. 57.

    in JDK’s Object.equals(obj)

  58. 58.

    in JFace’s ViewerColumn

  59. 59.

    in JFace’s IDocumentAdapter

  60. 60.

    in JDK’s Format

  61. 61.

    in JFace’s AbstractInformationControl.create()

  62. 62.

    in JDK’s SecurityManager.checkAccess(t)

  63. 63.

    in JFace’s ErrorDialog

  64. 64.

    in JDK’s Condition

  65. 65.

    in JDK’s PrivilegedExceptionAction.run()

  66. 66.

    in JDK’s ClassLoader.getSystemClassLoader()

  67. 67.

    in JDK’s KeyStore.Builder.getProtectionParameter(alias)

  68. 68.

    in Apache Commons Collections’ ComparatorChain.ComparatorChain()

  69. 69.

    in JDK’s Inflater.setInput(b)

  70. 70.

    in JFace’s JFaceResources.setFontRegistry(registry)

  71. 71.

    in JDK’s ActivationGroup.createGroup(id, desc, incarnation)

  72. 72.

    in JDK’s Socket.setReceiveBufferSize(size)

  73. 73.

    in SWT’s Event.gc

  74. 74.

    in JDK’s File.delete()

  75. 75.

    in JDK’s Acl.removeEntry(caller, entry)

  76. 76.

    in JDK’s Object.wait(timeout)

  77. 77.

    in JFace’s DefaultAutoIndentStrategy

  78. 78.

    in JDK’s StringBuffer

  79. 79.

    in JFace’s ContentViewer.setInput(input)

  80. 80.

    in JDK’s HashMap

  81. 81.

    in Apache Commons Collections’ FixedSizeMap

  82. 82.

    in JDK’s BeanContextSupport.addAll(c)

  83. 83.

    in JDK’s CertStoreSpi

  84. 84.

    in JDK’s SuppressWarnings.value()

  85. 85.

    in JDK’s Statement.executeBatch()

  86. 86.

    See http://en.wikipedia.org/wiki/Abundance_(ecology).

  87. 87.

    in JFace OwnerDrawLabelProvider.erase(event, element)

  88. 88.

    in JDK’s Logger.setParent(parent)

  89. 89.

    We exclusively focus on API directives and not on other concerns of API documentation (audience, writing process, etc.). For a comprehensive discussion of these other concerns, we refer to the excellent paper written by a senior technical writer at Sun: “API documentation from source code comments: a case study of Javadoc” (Kramer 1999).


  1. Alexander C, Ishikawa S, Silverstein M (1977) A pattern language: towns, buildings, construction. Oxford University Press

  2. Arnout K, Meyer B (2003) Uncovering hidden contracts: the .net example. Comput 36:48–55

    Article  Google Scholar 

  3. Beckman NE, Kim D, Aldrich J (2011) An empirical study of object protocols in the wild. In: Proceedings of the European conference on object-oriented programming. Springer.

  4. Bierhoff K, Beckman NE, Aldrich J (2009) Practical api protocol checking with access permissions. In: Proceedings of the 23rd European conference on object-oriented programming, Genoa. Springer, Berlin, Heidelberg, pp 195–219

    Google Scholar 

  5. Bloch J (2008) Effective java, 2nd edn. Addisson-Wesley

  6. Bokowski B (2008) Java API design. In: Tutorial at the EclipseCon conference

  7. Bokowski B, Arthorne J, des Rivières J (2006) Designing eclipse APIs. In: Tutorial at the EclipseCon conference

  8. Bradner S (1997) Key words for use in rfcs to indicate requirement levels. Technical report, IETF

  9. Brooks F (1995) The mythical man-month: essays on software engineering. Addison-Wesley

  10. Bruch M, Mezini M, Monperrus M (2010) Mining subclassing directives to improve framework reuse. In: Proceedings of the 7th IEEE working conference on mining software repositories. IEEE

  11. Clarke S (2004) Measuring API usability. Dr Dobb’s J 29(5):1–5

    Google Scholar 

  12. Cockburn A (2000) Writing effective use cases, 1st edn. Addison-Wesley Longman Publishing Co, Inc

  13. Dekel U (2009) Increasing awareness of delocalized information to facilitate API usage. PhD thesis, Carnegie Mellon University

  14. Dekel U, Herbsleb JD (2009) Improving API documentation usability with knowledge pushing. In: Proceedings of the 31st international conference on software engineering. IEEE Computer Society, pp 320–330

  15. Easterbrook S, Singer J, Storey M-A, Damian D (2007) Selecting empirical methods for software engineering research

  16. Ellis B, Stylos J, Myers B (2007) The factory pattern in api design: a usability evaluation. In: Proceedings of the 29th International Conference on Software Engineering, ICSE ’07. IEEE Computer Society, pp 302–312

  17. Gamma E, Helm R, Johnson RE, Vlissides J (1995) Design patterns: elements of reusable object-oriented software. Addison-Wesley

  18. Hovemeyer D, Spacco J, Pugh W (2005) Evaluating and tuning a static analysis to find null pointer bugs. In: Proceedings of the 6th ACM SIGPLAN-SIGSOFT workshop on program analysis for software tools and engineering, PASTE ’05. ACM, New York, NY, USA, pp 13–19

    Google Scholar 

  19. Jiang ZM, Hassan AE (2006) Examining the evolution of code comments in postgresql. In: Proceedings of the 2006 international workshop on mining software repositories. ACM, pp 179–180

  20. Kramer D (1999) API documentation from source code comments: a case study of javadoc. In: Proceedings of the 17th international conference on computer documentation. ACM, pp 147–153

  21. Monperrus M, Eichberg M, Tekes E, Mezini M (2011) Companion web page for “What should developers be aware of? An empirical study on the directives of API documentation”. http://www.monperrus.net/martin/api-directives. Last accessed: 12 Oct 2011

  22. Nanda MG, Sinha S (2009) Accurate interprocedural null-dereference analysis for java. In: Proceedings of the 31st international conference on software engineering. IEEE Computer Society, pp 133–143

  23. Padioleau Y, Tan L, Zhou Y (2009) Listening to programmers? Taxonomies and characteristics of comments in operating system code. In: Proceedings of the 31st international conference on software engineering. IEEE, pp 331–341

  24. Parnin C, Treude C (2011) Measuring API documentation on the Web. In: Proceeding of the 2nd international workshop on Web 2.0 for software engineering, Web2SE ’11. ACM, pp 25–30

  25. Robillard MP (2009) What makes APIs hard to learn? Answers from developers. IEEE Softw 26(6):27–34

    Article  Google Scholar 

  26. Shi L, Zhong H, Xie T, Li M (2011) An empirical study on evolution of api documentation. In: Proceedings of the 14th international conference on Fundamental Approaches to Software Engineering, FASE’11/ETAPS’11. Springer, pp 416–431

  27. Singh R, Mangat NS (1996) Elements of survey sampling. Springer

  28. Smith KA, Kramer D (2003) Requirements for writing java API specifications. Technical report, Sun

  29. Storey MA, Ryall J, Bull RI, Myers D, Singer J (2008) TODO or to bug. In: Proceedings of the ACM/IEEE 30th international conference on software engineering. IEEE, pp 251–260

  30. Stylos J (2009) Making APIs more usable with improved API designs, documentation and tools. PhD thesis, Carnegie Mellon University

  31. Stylos J, Myers BA (2008) The implications of method placement on API learnability. In: Proceedings of the 16th ACM SIGSOFT international symposium on foundations of software engineering, SIGSOFT ’08/FSE-16. ACM, New York, NY, USA, pp 105–112

    Google Scholar 

  32. Stylos J, Graf B, Busse DK, Ziegler C, Ehret R, Karstens J (2008) A case study of api redesign for improved usability. In: IEEE symposium on visual languages and human-centric computing, pp 189–192

  33. Stylos J, Faulring A, Yang Z, Myers BA (2009a) Improving API documentation using API usage information. In: Proceedings of the IEEE symposium on Visual Languages and Human-Centric Computing (VL/HCC’09)

  34. Stylos J, Myers BA, Yang Z (2009b) Jadeite: improving API documentation using usage information. In: Proceedings of the 27th international conference on human factors in computing systems. ACM, pp 4429–4434

  35. Tan L, Yuan D, Krishna G, Zhou Y (2007) icomment: bugs or bad comments. In: Proceedings of the symposium on operating systems principles

  36. Thummalapenta S, Xie T (2008) Spotweb: detecting framework hotspots and coldspots via mining open source code on the Web. In: Proceedings of the international conference on automated software engineering, pp 327–336

  37. Tulach J (2008) Practical API design: confessions of a java framework architect. Apress

  38. Watson RB (2009) Improving software API usability through text analysis: a case study. In: Proceedings of the professional communication conference, pp 1–7

  39. Ying ATT, Wright JL, Abrams S (2005) Source code that talks: an exploration of Eclipse task comments and their implication to repository mining. In: Proceedings of mining software repositories. ACM, pp 1–5

  40. Zhong H, Zhang L, Xie T, Mei H (2009) Inferring resource specifications from natural language API documentation. In: Proceedings of the international conference on automated software engineering. IEEE Computer Society, pp 307–318

Download references

Author information



Corresponding author

Correspondence to Martin Monperrus.

Electronic Supplementary Material

Below is the link to the electronic supplementary material.

(BZ2 2.62 MB)

(BZ2 1.20 MB)

(BZ2 119 KB)

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Monperrus, M., Eichberg, M., Tekes, E. et al. What should developers be aware of? An empirical study on the directives of API documentation. Empir Software Eng 17, 703–737 (2012). https://doi.org/10.1007/s10664-011-9186-4

Download citation


  • Application Program Interface
  • Syntactic Pattern
  • Library Method
  • Client Code
  • Comprehensive Taxonomy