Programming by Examples: Applications, Algorithms, and Ambiguity Resolution

Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9706)


99 % of computer end users do not know programming, and struggle with repetitive tasks. Programming by Examples (PBE) can revolutionize this landscape by enabling users to synthesize intended programs from example based specifications. A key technical challenge in PBE is to search for programs that are consistent with the examples provided by the user. Our efficient search methodology is based on two key ideas: (i) Restriction of the search space to an appropriate domain-specific language that offers balanced expressivity and readability (ii) A divide-and-conquer based deductive search paradigm that inductively reduces the problem of synthesizing a program of a certain kind that satisfies a given specification into sub-problems that refer to sub-programs or sub-specifications. Another challenge in PBE is to resolve the ambiguity in the example based specification. We will discuss two complementary approaches: (a) machine learning based ranking techniques that can pick an intended program from among those that satisfy the specification, and (b) active-learning based user interaction models. The above concepts will be illustrated using FlashFill, FlashExtract, and FlashRelate—PBE technologies for data manipulation domains. These technologies, which have been released inside various Microsoft products, are useful for data scientists who spend 80 % of their time wrangling with data. The Microsoft PROSE SDK allows easy construction of such technologies.


Ambiguity Resolution Reduction Logic Synthesis Algorithm Intended Program Search Methodology 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. 1.
    Alur, R., Bodik, R., Juniwal, G., Martin, M.M., Raghothaman, M., Seshia, S.A., Singh, R., Solar-Lezama, A., Torlak, E., Udupa, A.: Syntax-guided synthesis. In: FMCAD (2013)Google Scholar
  2. 2.
    Barowy, D.W., Gulwani, S., Hart, T., Zorn, B.G.: FlashRelate: extracting relational data from semi-structured spreadsheets using examples. In: PLDI (2015)Google Scholar
  3. 3.
    Cypher, A. (ed.): Watch What I Do: Programming by Demonstration. MIT Press, Cambridge (1993)Google Scholar
  4. 4.
    Gulwani, S.: Dimensions in program synthesis. In: PPDP (2010)Google Scholar
  5. 5.
    Gulwani, S.: Automating string processing in spreadsheets using input-output examples. In: POPL (2011)Google Scholar
  6. 6.
    Gulwani, S.: Programming by examples (and its applications in data wrangling). In: Esparza, J., Grumberg, O., Sickert, S. (eds.) Verification and Synthesis of Correct and Secure Systems. IOS Press (2016)Google Scholar
  7. 7.
    Gulwani, S., Harris, W., Singh, R.: Spreadsheet data manipulation using examples. Commun. ACM (2012)Google Scholar
  8. 8.
    Gulwani, S., Jha, S., Tiwari, A., Venkatesan, R.: Synthesis of loop-free programs. In: PLDI (2011)Google Scholar
  9. 9.
    Gulwani, S., Marron, M.: NLyze: interactive programming by natural language for spreadsheet data analysis and manipulation. In: SIGMOD (2014)Google Scholar
  10. 10.
    Harris, W.R., Gulwani, S.: Spreadsheet table transformations from examples. In: PLDI (2011)Google Scholar
  11. 11.
    Lau, T.: Why PBD systems fail: Lessons learned for usable AI. In: CHI 2008 Workshop on Usable AI (2008)Google Scholar
  12. 12.
    Le, V., Gulwani, S.: FlashExtract: a framework for data extraction by examples. In: PLDI (2014)Google Scholar
  13. 13.
    Le, V., Gulwani, S., Smartsynth, Z.: Synthesizing smartphone automation scripts from natural language. In: MobiSys (2013)Google Scholar
  14. 14.
    Leung, A., Sarracino, J., Lerner, S.: Interactive parser synthesis by example. In: PLDI (2015)Google Scholar
  15. 15.
    Lieberman, H.: Your Wish Is My Command: Programming by Example. Morgan Kaufmann, San Francisco (2001)Google Scholar
  16. 16.
    Mayer, M., Soares, G., Grechkin, M., Le, V., Marron, M., Polozov, O., Singh, R., Zorn, B., Gulwani, S.: User interaction models for disambiguation in programming by example. In: UIST (2015)Google Scholar
  17. 17.
    Meng, N., Kim, M., McKinley, K.S.: LASE: locating and applying systematic edits by learning from examples. In: ICSE (2013)Google Scholar
  18. 18.
    Polozov, O., Gulwani, S.: FlashMeta: a framework for inductive program synthesis. In: OOPSLA (2015).
  19. 19.
    Raza, M., Gulwani, S., Milic-Frayling, N.: Programming by example using least general generalizations. In: AAAI (2014)Google Scholar
  20. 20.
    Shen, Y., Chakrabarti, K., Chaudhuri, S., Ding, B., Novik, L.: Discovering queries based on example tuples. In: SIGMOD (2014)Google Scholar
  21. 21.
    Singh, R., Gulwani, S.: Learning semantic string transformations from examples. PVLDB 5, 740–751 (2012)Google Scholar
  22. 22.
    Singh, R., Gulwani, S.: Synthesizing number transformations from input-output examples. In: Madhusudan, P., Seshia, S.A. (eds.) CAV 2012. LNCS, vol. 7358, pp. 634–651. Springer, Heidelberg (2012)CrossRefGoogle Scholar
  23. 23.
    Singh, R., Gulwani, S.: Predicting a correct program in programming by example. In: Kroening, D., Păsăreanu, C.S. (eds.) CAV 2015. LNCS, vol. 9206, pp. 398–414. Springer, Heidelberg (2015)CrossRefGoogle Scholar
  24. 24.
    Singh, R., Gulwani, S.: Transforming spreadsheet data types using examples. In: POPL (2016)Google Scholar
  25. 25.
    Solar-Lezama, A.: Program Synthesis by Sketching. Ph.D. thesis, UC Berkeley (2008)Google Scholar
  26. 26.
    Srivastava, S., Gulwani, S., Foster, J.S.: From program verification to program synthesis. In: POPL (2010)Google Scholar
  27. 27.
    Udupa, A., Raghavan, A., Deshmukh, J.V., Mador-Haim, S., Martin, M.M.K., Alur, R.: TRANSIT: specifying protocols with concolic snippets. In: PLDI (2013)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2016

Authors and Affiliations

  1. 1.Microsoft CorporationRedmondUSA

Personalised recommendations