Abstract
Web-based applications are one of the most widely used types of software, and have become the backbone of many e-commerce and communications businesses. These applications are often mission-critical for many organizations, motivating their precise validation. Although regression testing has been widely used to gain confidence in the reliability of software by providing information about the quality of an application, it has suffered limited use in this domain due to the frequent nature of updates to websites and the difficulty of automatically comparing test case output. We present techniques to address these challenges in regression testing web-based applications. Without precise comparators, test cases that fail due to benign program evolutions must be manually inspected. Our approach harnesses the inherent similarities between unrelated web-based applications to provide fully automated solutions to reduce the number of such false positives, while simultaneously returning true faults. By applying a model derived from regression testing other programs, our approach can predict which test cases merit human inspection. Our method is 2.5 to 50 times as accurate as current industrial practice, but requires no user annotations.
Similar content being viewed by others
References
Apache Click. http://www.incubator.apache.org/click/ (2008)
A7soft jexamxml is a java based command line xml diff tool for comparing and merging xml documents. http://www.a7soft.com/jexamxml.html (2009)
Al-Ekram, R., Adma, A., Baysal, O.: diffX: an algorithm to detect changes in multi-version XML documents. In: Conference of the Centre for Advanced Studies on Collaborative Research, pp. 1–11. IBM Press, USA (2005)
Benedikt, M., Freire, J., Godefroid, P.: Veriweb: automatically testing dynamic web sites. In: World Wide Web Conference, May (2002)
Binder R.V.: Testing Object-Oriented Systems: Models, Patterns, and Tools. Addison-Wesley, Reading (1999)
Binkley, D.: Using semantic differencing to reduce the cost of regression testing. In: International Conference on Software Maintenance, pp. 41–50 (1992)
Binkley D.: Semantics guided regression test cost reduction. IEEE Trans. Softw. Eng. 23(8), 498–516 (1997)
Boehm B., Basili V.: Software defect reduction. IEEE Comput. Innov. Technol. Comput. Prof. 34(1), 135–137 (2001)
Chen, T.Y., Kuo, F.-C., Merkel, R.: On the statistical properties of the f-measure. In: International Conference on Quality Software, pp. 146–153 (2004)
Dobolyi, K., Weimer, W.: Harnessing web-based application similarities to aid in regression testing. In: 20th International Symposium on Software Reliability Engineering, November (2009)
Dobolyi, K., Weimer, W.: Modeling consumer-perceived web application fault severities for testing. Technical report, University of Virginia (2009)
Dobolyi, K., Weimer, W,: Addressing high severity faults in web application testing. In: The IASTED International Conference on Software Engineering, February (2010)
Elbaum, S., Karre, S., Rothermel, G.: Improving web application testing with user session data. In: International Conference on Software Engineering (2003)
Ellims, M.,Ince, D., Petre, M.: The csaw c mutation tool: initial results, pp. 185–192 (2007)
GCC-XML. http://www.gccxml.org/HTML/Index.html (2008)
Jean Harrold M., Gupta R., Soffa M.L.: A methodology for controlling the size of a test suite. ACM Trans. Softw. Eng. Methodol. 2(3), 270–285 (1993)
Hieatt E., Mee R.: Going faster: testing the web application. IEEE Softw. 19(2), 60–65 (2002)
Hooimeijer, P., Weimer, W.: Modeling bug report quality. In: Automated Software Engineering, pp. 34–43 (2007)
Kapfhammer, G.M.: Software testing. In: The Computer Science Handbook, ch. 105. CRC Press, Boca Raton (2004)
Karre S.: Leveraging user-session data to support web application testing. IEEE Trans. Softw. Eng. 31(3), 187–202 (2005)
Knight, J.C., Ammann, P.: An experimental evaluation of simple methods for seeding program errors. In: International Conference on Software Engineering, pp. 337–342 (1985)
Kohavi R.: A study of cross-validation and bootstrap for accuracy estimation and model selection. Int. Joint Conf. Artif. Intell. 14(2), 1137–1145 (1995)
Lee, S.C.L., Offutt, J.: Generating test cases for xml-based web component interactions using mutation analysis. In: International Symposium on Software Reliability Engineering, p. 200 (2001)
Di Lucca, G., Fasolino, A., Faralli, F., de Carlini, U.: Testing web applications. In: International Conference on Software Maintenance, p. 310 (2002)
Memon, A., Banerjee, I., Hashmi, N., Nagarajan, A.: DART: a framework for regression testing “nightly/daily builds” of GUI applications. In: International Conference on Software Maintenance (2003)
Meszaros, G.: Agile regression testing using record and playback. In: Object-oriented Programming, Systems, Languages, and Applications, pp. 353–360 (2003)
Neamtiu I., Foster J.S., Hicks M.: Understanding source code evolution using abstract syntax tree matching. SIGSOFT Softw. Eng. Notes 30(4), 1–5 (2005)
Offutt J.: Quality attributes of web software applications. IEEE Softw. 19(2), 25–32 (2002)
Open-realty. http://www.open-realty.org/ (2010)
Prestashop free open source e-commerce software for web 2.0. http://www.prestashop.com/ (2010)
Pertet, S., Narsimhan, P.: Causes of failures in web applications. Technical Report CMU-PDL-05-109, Carnegie Mellon University, December (2005)
Pressman R.S.: What a tangled web we weave [web engineering]. IEEE Softw. 17(1), 18–21 (2000)
Raghavan, S., Rohana, R., Leon, D., Podgurski, A., Augustine, V.: Dex: a semantic-graph differencing tool for studying changes in large code bases. In: International Conference on Software Maintenance (2004)
Ramamoothy C.V., Tsai W.-T.: Advances in software engineering. IEEE Comput. 29(10), 47–58 (1996)
Ricca F., Tonella P.: Testing processes of web applications. Ann. Softw. Eng. 14(1–4), 93–114 (2002)
Rosenberg, S.: What you don’t know can cost you millions, http://www.cxoamerica.com/pastissue/article.asp?art=270091&issue=202 (2009)
Rothermel G., Untch R.J., Chu C.: Prioritizing test cases for regression testing. IEEE Trans. Softw. Eng. 27(10), 929–948 (2001)
Salton G., McGill M.J.: Introduction to Modern Information Retrieval. McGraw-Hill, New York (1986)
Seacord R.C., Plakosh D., Lewis G.A.: Modernizing Legacy Systems: Software Technologies, Engineering Process and Business Practices. Addison-Wesley Longman Publishing Co., Inc., Boston (2003)
Silva, L.M.: Comparing error detection techniques for web applications: an experimental study. In: Network Computing and Applications, pp. 144–151 (2008)
Sneed, H.M.: Testing a web application. In: Workshop on Web Site Evolution, pp. 3–10 (2004)
Soechting, E., Dobolyi, K., Weimer, W.: Syntactic regression testing for tree-structured output. In: International Symposium on Web Systems Evolution, September (2009)
Soliman, F., Youssef, M.A.: Internet-based e-commerce and its impact on manufacturing and business operations. In: Industrial Management and Data Systems (2003)
Sprenkle, S., Gibson, E., Sampath, S., Pollock, L.: Automated replay and failure detection for web applications. In: Automated Software Engineering (2005)
Sprenkle, S., Hill, E., Pollock, L.: Learning effective oracle comparator combinations for web applications. In: International Conference on Quality Software (2007)
Sprenkle, S., Pollock, L., Esquivel, H., Hazelwood, B., Ecott, S.: Automated oracle comparators for testing web applications. In: International Symposium on Reliability Engineering, pp. 117–126 (2007)
Sutherland J.: Business objects in corporate information systems. ACM Comput. Surv. 27(2), 274–276 (1995)
txt2html—text to HTML converter. http://www.txt2html.sourceforge.net/. Technical report (2008)
van Wyk K.R., McGraw G.: Bridging the gap between software development and information security. IEEE Secur. Privacy 3, 75–79 (2005)
Vanilla—free, open-source forum software. http://vanillaforums.org/ (2010)
Vokolos, F.I., Frankl, P.G.: Pythia: a regression test selection tool based on textual differencing. In: Reliability, Quality and Safety of Software-Intensive Systems, pp. 3–21 (1997)
Vqwiki open source project. http://www.vqwiki.org/ (2008)
Walcott, K.R., Soffa, M.L., Kapfhammer, G.M., Roos, R.S.: Timeaware test suite prioritization. In: International Symposium on Software Testing and Analysis (2006)
Williamson, L.: IBM Rational software analyzer: beyond source code. In: Rational Software Developer Conference. http://www-07.ibm.com/in/events/rsdc2008/presentation2.html, June (2008)
Xing Z., Stroulia E.: Differencing logical uml models. Autom. Softw. Eng. 14(2), 215–259 (2007)
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Dobolyi, K., Soechting, E. & Weimer, W. Automating regression testing using web-based application similarities. Int J Softw Tools Technol Transfer 13, 111–129 (2011). https://doi.org/10.1007/s10009-010-0170-x
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10009-010-0170-x