Skip to main content
Book cover

International Symposium on Search Based Software Engineering

SSBSE 2021: Search-Based Software Engineering pp 9–25Cite as

Generating Failing Test Suites for Quantum Programs With Search

Part of the Lecture Notes in Computer Science book series (LNPSE,volume 12914)

Abstract

Testing quantum programs requires systematic, automated, and intelligent methods due to their inherent complexity, such as their superposition and entanglement. To this end, we present a search-based approach, called Quantum Search-Based Testing (QuSBT), for automatically generating test suites of a given size depending on available testing budget, with the aim of maximizing the number of failing test cases in the test suite. QuSBT consists of definitions of the problem encoding, failure types, test assessment with statistical tests, fitness function, and test case generation with a Genetic Algorithm (GA). To empirically evaluate QuSBT, we compared it with Random Search (RS) by testing six quantum programs. We assessed the effectiveness of QuSBT and RS with 30 carefully designed faulty versions of the six quantum programs. Results show that QuSBT provides a viable solution for testing quantum programs, and achieved a significant improvement over RS in 87% of the faulty programs, and no significant difference in the rest of 13% of the faulty programs.

Keywords

  • Quantum programs
  • Software testing
  • Genetic algorithms

QuSBT is supported by the National Natural Science Foundation of China under Grant No. 61872182 and Qu-Test (Project#299827) funded by Research Council of Norway. Paolo Arcaini is supported by ERATO HASUO Metamathematics for Systems Design Project (No. JPMJER1603), JST. Funding Reference number: 10.13039/501100009024 ERATO.

This is a preview of subscription content, access via your institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • DOI: 10.1007/978-3-030-88106-1_2
  • Chapter length: 17 pages
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
eBook
USD   44.99
Price excludes VAT (USA)
  • ISBN: 978-3-030-88106-1
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
Softcover Book
USD   59.99
Price excludes VAT (USA)
Learn about institutional subscriptions
Fig. 1.
Fig. 2.
Fig. 3.
Fig. 4.

Notes

  1. 1.

    Note that \(I\) and \(O\) do not need to be disjoint, i.e., an input qubit can also be an output qubit. Moreover, there could also be qubits that are neither inputs nor outputs, i.e., \(I \cup O \subseteq Q \).

  2. 2.

    In this case, we directly set \(\mathtt {fail} _{\textit{wodf}}\) (see Eq. 1 later) to false.

  3. 3.

    Note that the assessment for wodf is done only if the assessment for uof did not reveal any failure. If this is the case, it is guaranteed that the program returned outputs only from \(j_1, \ldots , j_k\), i.e., those having their expected probabilities being non-zero. Therefore, it is guaranteed that each returned output is considered in one of the counts \(c_{j_1}, \ldots , c_{j_k}\).

  4. 4.

    Note that a quantum program can still be deterministic for some given inputs.

  5. 5.

    https://github.com/Simula-COMPLEX/qusbt/.

  6. 6.

    Note that we manually approximated the value of programs with 1024 inputs values. Indeed, the correct number of tests would be \(\left\lceil {0.05 \cdot 1024}\right\rceil = 52\).

References

  1. Agresti, A.: An Introduction to Categorical Data Analysis, 3 edn. Wiley-Blackwell (2019)

    Google Scholar 

  2. Ali, S., Arcaini, P., Wang, X., Yue, T.: Assessing the effectiveness of input and output coverage criteria for testing quantum programs. In: 2021 14th IEEE Conference on Software Testing, Verification and Validation (ICST), pp. 13–23 (2021). https://doi.org/10.1109/ICST49551.2021.00014

  3. Arcuri, A., Briand, L.: A practical guide for using statistical tests to assess randomized algorithms in software engineering. In: Proceedings of the 33rd International Conference on Software Engineering, ICSE 2011, pp. 1–10. ACM, New York (2011)

    Google Scholar 

  4. Arcuri, A., Fraser, G.: Parameter tuning or default values? An empirical investigation in search-based software engineering. Empir. Softw. Eng. 18, 594–623 (2013)

    CrossRef  Google Scholar 

  5. Benítez-Hidalgo, A., Nebro, A.J., García-Nieto, J., Oregi, I., Del Ser, J.: jMetalPy: a Python framework for multi-objective optimization with metaheuristics. Swarm Evol. Comput. 51, 100598 (2019)

    CrossRef  Google Scholar 

  6. Gimeno-Segovia, M., Harrigan, N., Johnston, E.: Programming Quantum Computers: Essential Algorithms and Code Samples. O’Reilly Media, Inc., Newton (2019)

    Google Scholar 

  7. Honarvar, S., Mousavi, M.R., Nagarajan, R.: Property-based testing of quantum programs in Q#. In: Proceedings of the IEEE/ACM 42nd International Conference on Software Engineering Workshops, ICSEW 2020, pp. 430–435. Association for Computing Machinery, New York (2020)

    Google Scholar 

  8. Huang, Y., Martonosi, M.: QDB: from quantum algorithms towards correct quantum programs. In: 9th Workshop on Evaluation and Usability of Programming Languages and Tools (PLATEAU 2018). OpenAccess Series in Informatics (OASIcs), vol. 67, pp. 4:1–4:14. Schloss Dagstuhl-Leibniz-Zentrum fuer Informatik, Dagstuhl (2019)

    Google Scholar 

  9. Li, G., Zhou, L., Yu, N., Ding, Y., Ying, M., Xie, Y.: Projection-based runtime assertions for testing and debugging quantum programs. Proc. ACM Program. Lang. 4(OOPSLA), 1–29 (2020)

    Google Scholar 

  10. Liu, J., Byrd, G.T., Zhou, H.: Quantum circuits for dynamic runtime assertions in quantum computation. In: Proceedings of the Twenty-Fifth International Conference on Architectural Support for Programming Languages and Operating Systems, pp. 1017–1030 (2020)

    Google Scholar 

  11. Miranskyy, A., Zhang, L.: On testing quantum programs. In: 2019 IEEE/ACM 41st International Conference on Software Engineering: New Ideas and Emerging Results (ICSE-NIER), pp. 57–60 (2019)

    Google Scholar 

  12. Wang, J., Ma, F., Jiang, Y.: Poster: fuzz testing of quantum program. In: 2021 14th IEEE Conference on Software Testing, Verification and Validation (ICST), pp. 466–469 (2021). https://doi.org/10.1109/ICST49551.2021.00061

  13. Wille, R., Van Meter, R., Naveh, Y.: IBM’s Qiskit tool chain: working with and developing for real quantum computers. In: 2019 Design, Automation Test in Europe Conference Exhibition (DATE), pp. 1234–1240 (2019)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Nanjing University of Aeronautics and Astronautics, Nanjing, China

    Xinyi Wang & Tao Yue

  2. National Institute of Informatics, Tokyo, Japan

    Paolo Arcaini

  3. Simula Research Laboratory, Fornebu, Norway

    Tao Yue & Shaukat Ali

Authors
  1. Xinyi Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Paolo Arcaini
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Tao Yue
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Shaukat Ali
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Xinyi Wang .

Editor information

Editors and Affiliations

  1. Massachusetts Institute of Technology, Cambridge, MA, USA

    Prof. Una-May O'Reilly

  2. Delft University of Technology, Delft, The Netherlands

    Ph.D. Xavier Devroey

Rights and permissions

Reprints and Permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this paper

Verify currency and authenticity via CrossMark

Cite this paper

Wang, X., Arcaini, P., Yue, T., Ali, S. (2021). Generating Failing Test Suites for Quantum Programs With Search. In: O'Reilly, UM., Devroey, X. (eds) Search-Based Software Engineering. SSBSE 2021. Lecture Notes in Computer Science(), vol 12914. Springer, Cham. https://doi.org/10.1007/978-3-030-88106-1_2

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-88106-1_2

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-88105-4

  • Online ISBN: 978-3-030-88106-1

  • eBook Packages: Computer ScienceComputer Science (R0)