Marcus Kessel
ABSTRACT
Test amplification approaches take a manually written set of tests (input/output mappings) and enhance their effectiveness for some clearly defined engineering goal such as detecting faults. Conceptually, they can either achieve this in a ``black box'' way using only the initial ``seed'' tests or in a ``white box'' way utilizing additional inputs such as the source code or specification of the software under test. However, no fully black box approach to test amplification is currently available even though they can be used to enhance white box approaches. In this paper we introduce a new approach that uses the seed tests to search for existing redundant implementations of the software under test and leverages them as oracles in the generation and evaluation of new tests. The approach can therefore be used as a stand alone black box test amplification method or in tandem with other methods. In this paper we explain the approach, describe its synergies with other approaches and provide some evidence for its practical feasibility.
- Miltiadis Allamanis, Earl T. Barr, Premkumar Devanbu, and Charles Sutton. 2018. A Survey of Machine Learning for Big Code and Naturalness. ACM Comput. Surv. 51, 4, Article 81 (July 2018), 37 pages. Google Scholar
Digital Library
- Paul Ammann and Jeff Offutt. 2008. Introduction to Software Testing (1 ed.). Cambridge University Press. Google ScholarDigital Library
- Earl T. Barr, Yuriy Brun, Premkumar Devanbu, Mark Harman, and Federica Sarro. 2014. The Plastic Surgery Hypothesis. In Proceedings of the 22Nd ACM SIGSOFT International Symposium on Foundations of Software Engineering (FSE 2014). ACM, New York, NY, USA, 306–317. Google ScholarDigital Library
- E. T. Barr, M. Harman, P. McMinn, M. Shahbaz, and S. Yoo. 2015. The Oracle Problem in Software Testing: A Survey. IEEE Transactions on Software Engineering 41, 5 (May 2015), 507–525.Google ScholarDigital Library
- B. Beckert and R. Hähnle. 2014. Reasoning and Verification: State of the Art and Current Trends. IEEE Intelligent Systems 29, 1 (Jan 2014), 20–29.Google Scholar
- Carl Boettiger. 2015. An Introduction to Docker for Reproducible Research. SIGOPS Oper. Syst. Rev. 49, 1 (Jan. 2015), 71–79. Google ScholarDigital Library
- 2723882Google Scholar
- Antonio Carzaniga, Alberto Goffi, Alessandra Gorla, Andrea Mattavelli, and Mauro Pezzè. 2014. Cross-checking Oracles from Intrinsic Software Redundancy. In Proceedings of the 36th International Conference on Software Engineering (ICSE 2014). ACM, New York, NY, USA, 931–942. Google ScholarDigital Library
- 2568287Google Scholar
- Maven Central. 2019. Website. http://search.maven.org (accessed 2019-05-01).Google Scholar
- A. C. d. Paula, E. Guerra, C. V. Lopes, H. Sajnani, and O. A. L. Lemos. 2016. An Exploratory Study of Interface Redundancy in Code Repositories. In 2016 IEEE 16th International Working Conference on Source Code Analysis and Manipulation (SCAM). 107–116.Google Scholar
- Benjamin Danglot, Oscar Vera-Perez, Zhongxing Yu, Martin Monperrus, and Benoit Baudry. 2018. A Snowballing Literature Study on Test Amplification. CoRR abs/1705.10692v2 (2018). arXiv: 1705.10692v2 http://arxiv.org/abs/1705.10692v2Google Scholar
- Benjamin Danglot, Oscar Luis Vera-Pérez, Benoit Baudry, and Martin Monperrus. 2018. Automatic Test Improvement with DSpot: a Study with Ten Mature Open-Source Projects. CoRR abs/1811.08330 (2018). arXiv: 1811.08330 http://arxiv.org/ abs/1811.08330Google Scholar
- Luca Della Toffola, Cristian Alexandru Staicu, and Michael Pradel. 2017. Saying ’Hi!’ is Not Enough: Mining Inputs for Effective Test Generation. In Proceedings of the 32Nd IEEE/ACM International Conference on Automated Software Engineering (ASE 2017). IEEE Press, Piscataway, NJ, USA, 44–49. http://dl.acm.org/citation. cfm?id=3155562.3155572 Google ScholarDigital Library
- Robert Dyer, Hoan Anh Nguyen, Hridesh Rajan, and Tien N. Nguyen. 2013. Boa: A Language and Infrastructure for Analyzing Ultra-large-scale Software Repositories. In Proceedings of the 2013 International Conference on Software Engineering (ICSE ’13). IEEE Press, Piscataway, NJ, USA, 422–431. http://dl.acm. org/citation.cfm?id=2486788.2486844 Google ScholarDigital Library
- D. E. Eckhardt, A. K. Caglayan, J. C. Knight, L. D. Lee, D. F. McAllister, M. A. Vouk, and J. P. J. Kelly. 1991. An experimental evaluation of software redundancy as a strategy for improving reliability. IEEE Transactions on Software Engineering 17, 7 (July 1991), 692–702. Google ScholarDigital Library
- Michael D. Ernst, Jeff H. Perkins, Philip J. Guo, Stephen McCamant, Carlos Pacheco, Matthew S. Tschantz, and Chen Xiao. 2007. The Daikon system for dynamic detection of likely invariants. Science of Computer Programming 69, 1 (2007), 35 – 45. Special issue on Experimental Software and Toolkits. Google ScholarDigital Library
- Gordon Fraser and Andrea Arcuri. 2011. EvoSuite: Automatic Test Suite Generation for Object-oriented Software. In Proceedings of the 19th ACM SIGSOFT A Platform for Diversity-Driven Test Amplification A-TEST ’19, August 26–27, 2019, Tallinn, Estonia Symposium and the 13th European Conference on Foundations of Software Engineering (ESEC/FSE ’11). ACM, New York, NY, USA, 416–419. 1145/2025113.2025179 Google ScholarDigital Library
- Gordon Fraser and Andrea Arcuri. 2014. A Large-Scale Evaluation of Automated Unit Test Generation Using EvoSuite. ACM Trans. Softw. Eng. Methodol. 24, 2, Article 8 (Dec. 2014), 42 pages. Google ScholarDigital Library
- G. Grahne and J. Zhu. 2005. Fast algorithms for frequent itemset mining using FP-trees. IEEE Transactions on Knowledge and Data Engineering 17, 10 (Oct 2005), 1347–1362. Google ScholarDigital Library
- Robert M. Hierons, Kirill Bogdanov, Jonathan P. Bowen, Rance Cleaveland, John Derrick, Jeremy Dick, Marian Gheorghe, Mark Harman, Kalpesh Kapoor, Paul Krause, Gerald Lüttgen, Anthony J. H. Simons, Sergiy Vilkomir, Martin R. Woodward, and Hussein Zedan. 2009. Using Formal Specifications to Support Testing. ACM Comput. Surv. 41, 2, Article 9 (Feb. 2009), 76 pages. Google ScholarDigital Library
- Oliver Hummel. 2008. Semantic component retrieval in software engineering. Ph.D. Dissertation. Universität Mannheim.Google Scholar
- O. Hummel, W. Janjic, and C. Atkinson. 2008. Code Conjurer: Pulling Reusable Software out of Thin Air. IEEE Software 25, 5 (Sep. 2008), 45–52. Google ScholarDigital Library
- Werner Janjic and Colin Atkinson. 2013. Utilizing Software Reuse Experience for Automated Test Recommendation. In Proceedings of the 8th International Workshop on Automation of Software Test (AST ’13). IEEE Press, Piscataway, NJ, USA, 100–106. http://dl.acm.org/citation.cfm?id=2662413.2662436 Google ScholarDigital Library
- Werner Janjic, Florian Barth, Oliver Hummel, and Colin Atkinson. 2011. Discrepancy Discovery in Search-enhanced Testing. In Proceedings of the 3rd International Workshop on Search-Driven Development: Users, Infrastructure, Tools, and Evaluation (SUITE ’11). ACM, New York, NY, USA, 21–24. 1985429.1985435 Google ScholarDigital Library
- Jenkins. 2019. Website. https://jenkins.io/. (accessed 2019-05-01).Google Scholar
- René Just. 2014. The Major Mutation Framework: Efficient and Scalable Mutation Analysis for Java. In Proceedings of the 2014 International Symposium on Software Testing and Analysis (ISSTA 2014). ACM, New York, NY, USA, 433–436. Google ScholarDigital Library
- Y. Ke, K. T. Stolee, C. L. Goues, and Y. Brun. 2015. Repairing Programs with Semantic Code Search (T). In 2015 30th IEEE/ACM International Conference on Automated Software Engineering (ASE). 295–306. 2015.60Google ScholarDigital Library
- Marcus Kessel and Colin Atkinson. 2016. Ranking software components for reuse based on non-functional properties. Information Systems Frontiers 18, 5 (01 Oct 2016), 825–853. Google ScholarDigital Library
- M. Kessel and C. Atkinson. 2018. Integrating Reuse into the Rapid, Continuous Software Engineering Cycle through Test-Driven Search. In 2018 IEEE/ACM 4th International Workshop on Rapid Continuous Software Engineering (RCoSE). 8–11. Google ScholarDigital Library
- Groovy Language. 2019. Website. https://http://groovy-lang.org/ (accessed 2019-05-01).Google Scholar
- Otávio Augusto Lazzarini Lemos, Sushil Krishna Bajracharya, Joel Ossher, Ricardo Santos Morla, Paulo Cesar Masiero, Pierre Baldi, and Cristina Videira Lopes. 2007. CodeGenie: Using Test-cases to Search and Reuse Source Code. In Proceedings of the Twenty-second IEEE/ACM International Conference on Automated Software Engineering (ASE ’07). ACM, New York, NY, USA, 525–526. Google ScholarDigital Library
- Otávio Augusto Lazzarini Lemos, Fabiano Cutigi Ferrari, Fábio Fagundes Silveira, and Alessandro Garcia. 2012.Google Scholar
- Development of Auxiliary Functions: Should You Be Agile? An Empirical Assessment of Pair Programming and Test-first Programming. In Proceedings of the 34th International Conference on Software Engineering (ICSE ’12). IEEE Press, Piscataway, NJ, USA, 529–539. http://dl.acm.org/citation.cfm?id=2337223.2337285 Google ScholarDigital Library
- Christopher D. Manning, Prabhakar Raghavan, and Hinrich Schütze. 2008. Introduction to Information Retrieval. Cambridge University Press. Google Scholar
- Maven. 2019. Website. http://maven.apache.org/ (accessed 2019-05-01).Google Scholar
- Carlos Pacheco, Shuvendu K. Lahiri, Michael D. Ernst, and Thomas Ball. 2007. Feedback-Directed Random Test Generation. In Proceedings of the 29th International Conference on Software Engineering (ICSE ’07). IEEE Computer Society, Washington, DC, USA, 75–84. Google ScholarDigital Library
- Andy Podgurski and Lynn Pierce. 1993. Retrieving Reusable Software by Sampling Behavior. ACM Trans. Softw. Eng. Methodol. 2, 3 (July 1993), 286–303. Google ScholarDigital Library
- Steven P. Reiss. 2009. Semantics-based Code Search. In Proceedings of the 31st International Conference on Software Engineering (ICSE ’09). IEEE Computer Society, Washington, DC, USA, 243–253. Google ScholarDigital Library
- 5070525Google Scholar
- M. Robillard, R. Walker, and T. Zimmermann. 2010. Recommendation Systems for Software Engineering. IEEE Software 27, 4 (July 2010), 80–86. org/10.1109/MS.2009.161 Google ScholarDigital Library
- Chanchal K. Roy, James R. Cordy, and Rainer Koschke. 2009. Comparison and Evaluation of Code Clone Detection Techniques and Tools: A Qualitative Approach. Sci. Comput. Program. 74, 7 (May 2009), 470–495. Google ScholarDigital Library
- S. Segura, G. Fraser, A. B. Sanchez, and A. Ruiz-CortÃľs. 2016. A Survey on Metamorphic Testing. IEEE Transactions on Software Engineering 42, 9 (Sep. 2016), 805–824.Google ScholarCross Ref
- Kathryn T. Stolee, Sebastian Elbaum, and Daniel Dobos. 2014. Solving the Search for Source Code. ACM Trans. Softw. Eng. Methodol. 23, 3, Article 26 (June 2014), 45 pages. Google ScholarDigital Library
- M. Suzuki, A. C. d. Paula, E. Guerra, C. V. Lopes, and O. A. L. Lemos. 2017. An Exploratory Study of Functional Redundancy in Code Repositories. In 2017 IEEE 17th International Working Conference on Source Code Analysis and Manipulation (SCAM). 31–40.Google Scholar
Index Terms
- A platform for diversity-driven test amplification
Recommendations
Diversity-driven unit test generation
AbstractThe goal of automated unit test generation tools is to create a set of test cases for the software under test that achieve the highest possible coverage for the selected test quality criteria. The most effective approaches for ...
Highlights- Study of novel diversity-driven test unit generation (DivGen) exploiting redundancy.
Automatic test amplification for executable models
MODELS '22: Proceedings of the 25th International Conference on Model Driven Engineering Languages and SystemsBehavioral models are important assets that must be thoroughly verified early in the design process. This can be achieved with manually-written test cases that embed carefully hand-picked domain-specific input data. However, such test cases may not ...
Developer-centric test amplification: The interplay between automatic generation human exploration
AbstractAutomatically generating test cases for software has been an active research topic for many years. While current tools can generate powerful regression or crash-reproducing test cases, these are often kept separately from the maintained test ...
Comments