Skip to main content
SpringerLink
Log in

Math Search for the Masses: Multimodal Search Interfaces and Appearance-Based Retrieval

  • Conference paper
  • First Online:
Intelligent Computer Mathematics (CICM 2015)

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 9150))

Included in the following conference series:

Abstract

We summarize math search engines and search interfaces produced by the Document and Pattern Recognition Lab in recent years, and in particular the \(m_{in}\) math search interface and the Tangent search engine. Source code for both systems are publicly available. “The Masses” refers to our emphasis on creating systems for mathematical non-experts, who may be looking to define unfamiliar notation, or browse documents based on the visual appearance of formulae rather than their mathematical semantics.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    http://www.w3.org/Math/.

  2. 2.

    https://www.mathjax.org/.

  3. 3.

    Source code: http://www.cs.rit.edu/~dprl/Software.html.

  4. 4.

    http://www.isical.ac.in/~crohme/.

  5. 5.

    http://www.myscript.com/.

  6. 6.

    http://scikit-learn.org/.

  7. 7.

    https://lucene.apache.org/.

  8. 8.

    http://saskatoon.cs.rit.edu/tangent.

  9. 9.

    This approach was motivated by a ranking function that used sets of matching symbols and symbol pairs to greatly improve initial retrieval results [27].

References

  1. Aizawa, A., Kohlhase, M., Ounis, I., Schubotz, M.: NTCIR-11 Math-2 task overview. In: Proceedings of the 11th NTCIR Conference, Tokyo, Japan, pp. 99–102, December 2014

    Google Scholar 

  2. Baker, J.B., Sexton, A.P., Sorge, V.: A linear grammar approach to mathematical formula recognition from PDF. In: Carette, J., Dixon, L., Coen, C.S., Watt, S.M. (eds.) Calculemus/MKM 2009. LNCS, vol. 5625, pp. 201–16. Springer, Heidelberg (2009)

    Chapter  Google Scholar 

  3. Blostein, D., Zanibbi, R.: Processing mathematical notation. In: Doermann, D., Tombre, K. (eds.) Handbook of Document Image Processing and Recognition, pp. 679–702. Springer, London (2014)

    Chapter  Google Scholar 

  4. Chatbri, H., Kwan, P.W., Kameyama, K.: A modular approach for query spotting in document images and its optimization using genetic algorithms. In: Proceedings of the IEEE Congress on Evolutionary Computation, Beijing, China, pp. 2085–2092, July 2014

    Google Scholar 

  5. Cordy, J.R.: The TXL source transformation language. Sci. Comput. Program. 61(3), 190–10 (2006)

    Article  MathSciNet  MATH  Google Scholar 

  6. Davila, K., Agarwal, A., Gaborski, R., Zanibbi, R., Ludi, S.: AccessMath: Indexing and retrieving video segments containing math expressions based on visual similarity. In: Proceedings of the IEEE Western New York Image Processing Workshop, Rochester, NY, pp. 14–17 (2013)

    Google Scholar 

  7. Davila, K., Ludi, S., Zanibbi, R.: Using off-line features and synthetic data for on-line handwritten math symbol recognition. In: Proceedings of the International Conference Frontiers in Handwriting Recognition, Crete, Greece, pp. 323–328 (2014)

    Google Scholar 

  8. Hiroya, H., Saito, H.: Partial-match retrieval with structure-reflected indices at the NTCIR-10 math task. In: Proceedings of the NII Testbeds and Community for Information Access Research, Tokyo, Japan, pp. 692–695, June 2013

    Google Scholar 

  9. Hu, L., Zanibbi, R.: HMM-based recognition of online handwritten mathematical symbols using segmental k-means initialization and a modified pen-up/down feature. In: Proceedings of the International Conference Document Analysis and Recognition, pp. 457–462 (2011)

    Google Scholar 

  10. Kamali, S., Tompa, F.W.: A new mathematics retrieval system. In: Proceedings of the 19th ACM International Conference on Information and Knowledge Management, CIKM 2010, pp. 1413–1416. ACM, New York (2010)

    Google Scholar 

  11. Kamali, S., Tompa, F.W.: Structural similarity search for mathematics retrieval. In: Carette, J., Aspinall, D., Lange, C., Sojka, P., Windsteiger, W. (eds.) CICM 2013. LNCS, vol. 7961, pp. 246–62. Springer, Heidelberg (2013)

    Chapter  Google Scholar 

  12. Kohlhase, M., Sucan, I.: A search engine for mathematical formulae. In: Calmet, J., Ida, T., Wang, D. (eds.) AISC 2006. LNCS (LNAI), vol. 4120, pp. 241–53. Springer, Heidelberg (2006)

    Chapter  Google Scholar 

  13. Lamport, L.: : A Document Preparation System. Addison-Wesley Reading, MA (1986)

    Google Scholar 

  14. Landy, D., Goldstone, R.: Formal notations are diagrams: Evidence from a production task. Mem. Cogn. 35(8), 2033–40 (2007)

    Article  Google Scholar 

  15. Landy, D., Goldstone, R.: How abstract is symbolic thought? J. Exp. Psychol. Learn. Mem. Cogn. 35(8), 720–33 (2007)

    Article  Google Scholar 

  16. Lin, X., Gao, L., Hu, X., Tang, Z., Xiao, Y., Liu, X.: A mathematics retrieval system for formulae in layout presentations. In: Proceedings of the ACM SIGIR, pp. 697–706 (2014)

    Google Scholar 

  17. MacLean, S., Labahn, G.: A new approach for recognizing handwritten mathematics using relational grammars and fuzzy sets. Int. J. Doc. Anal. Recogn. (IJDAR) 16(2), 1–25 (2012)

    Google Scholar 

  18. Miller, B.R., Youssef, A.: Technical aspects of the digital library of mathematical functions. Ann. Math. Artif. Intell. 38, 121–36 (2003)

    Article  MathSciNet  MATH  Google Scholar 

  19. Mouchére, H., Viard-Gaudin, C., Zanibbi, R., Garain, U.: ICFHR 2014 competition on recognition of on-line handwritten mathematical expressions (CROHME 2014). In: Proceedings of the International Conference Frontiers in Handwriting Recognition, Crete, Greece, pp. 791–796 (2014)

    Google Scholar 

  20. Nagy, G., Seth, S.: Hierarchical representation of optically scanned documents. In: Proceedings of the Seventh International Conference on Pattern Recognition, Montreal, Canada, pp. 347–349 (1984)

    Google Scholar 

  21. Nguyen, T.T., Hui, S.C., Chang, K.: A lattice-based approach for mathematical search using formal concept analysis. Expert Syst. Appl. 39(5), 5820–8 (2012)

    Article  Google Scholar 

  22. Pattaniyil, N., Zanibbi, R.: Combining TF-IDF text retrieval with an inverted index over symbol pairs in math expressions: The Tangent math search engine at NTCIR 2014. In: Proceedings of the 1st NII Testbeds and Community for Information access Research (NTCIR), Tokyo, Japan (2014) (online, p. 8)

    Google Scholar 

  23. Pollanen, M., Wisniewski, T., Yu, X.: Xpress: A novice interface for the real-time communication of mathematical expressions. In: Proceedings of the Workshop Mathematical User-Interfaces, Linz, Austria, June 2007

    Google Scholar 

  24. Rath, T., Manmatha, R.: Word spotting for historical documents. Int. J. Doc. Anal. Recogn. 9(2–4), 139–52 (2007)

    Article  Google Scholar 

  25. Reichenbach, M., Agarwal, A., Zanibbi, R.: Rendering expressions to improve accuracy of relevance assessment for math search. In: Proceedings of the ACM SIGIR, Gold Coast, Australia, pp. 851–854 (2014)

    Google Scholar 

  26. Sasarak, C., Hart, K., Pospesel, R., Stalnaker, D., Hu, L., LiVolsi, R., Zhu, S., Zanibbi, R.: \(m_{in}\): a multimodal web interface for web search. In: Symp. Human-Computer Interaction and Information Retrieval, Cambridge, MA, pp. (online, p. 4), Oct 2012

    Google Scholar 

  27. Schellenberg, T., Yuan, B., Zanibbi, R.: Layout-based substitution tree indexing and retrieval for mathematical expressions. In: Proceedings of the Document Recognition and Retrieval XVIII, pp. OI:1–8 (2012)

    Google Scholar 

  28. Schubotz, M.: Challenges of mathematical information retrieval in the NTCIR-11 Math Wikipedia Task. In: Proceedings of the SIGIR (2015, to appear)

    Google Scholar 

  29. Smithies, S., Novins, K., Arvo, J.: A handwriting-based equation editor. In: Proceedings of the Graphics Interface, Kingston, ON, June 1999

    Google Scholar 

  30. Sojka, P., Líška, M.: Indexing and searching mathematics in digital libraries. In: Davenport, J.H., Farmer, W.M., Urban, J., Rabe, F. (eds.) Calculemus/MKM 2011. LNCS, vol. 6824, pp. 228–43. Springer, Heidelberg (2011)

    Chapter  Google Scholar 

  31. Pavan Kumar, P., Agarwal, A., Bhagvati, C.: A structure based approach for mathematical expression retrieval. In: Sombattheera, C., Loi, N.K., Wankar, R., Quan, T. (eds.) MIWAI 2012. LNCS, vol. 7694, pp. 23–34. Springer, Heidelberg (2012)

    Chapter  Google Scholar 

  32. Stalnaker, D., Zanibbi, R.: Math expression retrieval using an inverted index over symbol pairs. In: Proceedings of the Document Recognition and Retrieval XXII. Proc. SPIE, San Francisco, USA, vol. 9402, pp. 940207–1:12, Feb 2015

    Google Scholar 

  33. Suzuki, M., Kanahori, T., Ohtake, N., Yamaguchi, K.: An integrated OCR software for mathematical documents and its output with accessibility. In: Miesenberger, K., Klaus, J., Zagler, W.L., Burger, D. (eds.) ICCHP 2004. LNCS, vol. 3118, pp. 648–55. Springer, Heidelberg (2004)

    Chapter  Google Scholar 

  34. Uchida, S., Nomura, A., Suzuki, M.: Quantitative analysis of mathematical documents. Int. J. Doc. Anal. Recogn. 7(4), 211–8 (2005)

    Article  Google Scholar 

  35. Wangari, K., Zanibbi, R., Agarwal, A.: Discovering real-world use cases for a multimodal math search interface. In: Proceedings of the ACM SIGIR, Gold Coast, Australia, pp. 947–950, July 2014

    Google Scholar 

  36. Youssef, A.S.: Methods of relevance ranking and hit-content generation in math search. In: Kauers, M., Kerber, M., Miner, R., Windsteiger, W. (eds.) MKM/CALCULEMUS 2007. LNCS (LNAI), vol. 4573, pp. 393–406. Springer, Heidelberg (2007)

    Chapter  Google Scholar 

  37. Zanibbi, R., Blostein, D.: Recognition and retrieval of mathematical expressions. Int. J. Doc. Anal. Recogn. (IJDAR) 15(4), 331–57 (2012)

    Article  Google Scholar 

  38. Zanibbi, R., Yu, L.: Math spotting: Retrieving math in technical documents using handwritten query images. In: Proceedings of the International Conference on Document Analysis and Recognition, Beijing, China, pp. 446–451, September 2011

    Google Scholar 

  39. Zanibbi, R., Yuan, B.: Keyword and image-based retrieval of mathematical expressions. In: Proceedings of the Document Recognition and Retrieval XVIII, pp. 78740I–78740I (2011)

    Google Scholar 

  40. Zanibbi, R., Blostein, D., Cordy, J.R.: Recognizing mathematical expressions using tree transformation. IEEE Trans. Pattern Anal. Mach. Intell. 24(11), 1455–1467 (2002)

    Article  Google Scholar 

  41. Zanibbi, R., Novins, K., Arvo, J., Zanibbi, K.: Aiding manipulation of handwritten mathematical expressions through style-preserving morphs. In: Proceedings of the Graphics Interface, Ottawa, ON, June 2001

    Google Scholar 

  42. Zhao, J., Kan, M.Y., Theng, Y.L.: Math information retrieval: user requirements and prototype implementation. In: JCDL 2008: Proceedings of the 8th ACM/IEEE-CS Joint Conference on Digital Libraries, pp. 187–196. ACM, New York (2008)

    Google Scholar 

  43. Zhu, S., Hu, L., Zanibbi, R.: Rotation-robust math symbol recognition and retrieval using outer contours and image subsampling. In: Proceedings of the Document Recognition and Retrieval XX, San Francisco, CA, pp. 5:1–5:12, Feb 2013

    Google Scholar 

Download references

Acknowledgements

We thank George Nagy, Maria Zemankova, Christian Viard-Gaudin, Harold Mouchère, Frank Tompa and Andrew Kane for helpful discussions. This material is based upon work supported by the National Science Foundation (USA) under Grant Numbers IIS-1016815 and HCC-1218801.

Author information

Authors and Affiliations

  1. Document and Pattern Recognition Laboratory, Department of Computer Science, Rochester Institute of Technology, Rochester, New York, 14623, USA

    Richard Zanibbi & Awelemdy Orakwue

Authors
  1. Richard Zanibbi
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Awelemdy Orakwue
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Richard Zanibbi .

Editor information

Editors and Affiliations

  1. School of Computer Science, University of Birmingham, Birmingham, United Kingdom

    Manfred Kerber

  2. McMaster University, Hamilton, Ontario, Canada

    Jacques Carette

  3. University of Innsbruck, Innsbruck, Austria

    Cezary Kaliszyk

  4. Jacobs University Bremen, Bremen, Germany

    Florian Rabe

  5. University of Birmingham, Birmingham, United Kingdom

    Volker Sorge

Rights and permissions

Reprints and permissions

Copyright information

© 2015 Springer International Publishing Switzerland

About this paper

Cite this paper

Zanibbi, R., Orakwue, A. (2015). Math Search for the Masses: Multimodal Search Interfaces and Appearance-Based Retrieval. In: Kerber, M., Carette, J., Kaliszyk, C., Rabe, F., Sorge, V. (eds) Intelligent Computer Mathematics. CICM 2015. Lecture Notes in Computer Science(), vol 9150. Springer, Cham. https://doi.org/10.1007/978-3-319-20615-8_2

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-20615-8_2

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-20614-1

  • Online ISBN: 978-3-319-20615-8

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation