An Effective Approach to High Strength Covering Array Generation in Combinatorial Testing, the latest research result of the research team of Prof. Zhou Jiantao with the College of Computer Science (College of Software) of IMU, has been officially accepted by the international top journal in the field of computer science- IEEE Transactions on Software Engineering and the preprinted version of the article has been published in IEEE Xplore(DOI: 10.1109/TSE.2023.3306461).

The research has chosen in the field of software testing to study the testing methods for high strength covering array generation which is highly theoretical and practical and whose research result may effectively accelerate the calculation process of candidate test cases, support the generation of the test case set with higher covering strength and is theoretically important for testing models and systems in such key fields as big data analysis, big model, cloud computing and internet of things.

Against the background of such applications as the new computing architecture and artificial intelligence, the combinatorial testing methods which can support higher covering strength needs to be used to ensure the higher reliability of large-scale and complex software. However, with the increasing complexity and parameters of the software under test, most of the methods of the combinatorial testcase generation only supports the generation of test cases with low covering strength(Fig. 2 and Fig. 3). Despite its wide research and application, the swarm-intelligence-based metaheuristic algorithms, needs to evaluate the strengths and weaknesses of the solutions by judging the combinatorial test covering of the candidate solutions. Moreover, due to the large number of candidate solutions and the need to upgrade the algorithm, it is hard for the swarm-intelligence-based metaheuristic algorithms to support a high covering strength.

In order to support the generation of the combination test cases with higher covering strength through metaheuristic algorithm, the article, based on the specially-designed data architecture, proposes the combination location methods which are universal and applicable to other search-based methods and presents the process of fitness calculation(Fig. 1).

Fig. 1 The process of fitness computation

The data in Table. 1 show that the proposed combination location methods and the methods for fitness computation, in comparison with the commonly used approaches for the generation of combinatorial test cases, have the smallest time complexity and the proposed data structure has the smallest space complexity.

Table. 1 Comparison of time complexity and space complexity

Besides, the article applies multi-learning strategies, including Lamarckian and Baldwinian learning, to multi-learning based quantum particle swarm optimization. According to the information feedback from the construction of covering arrays, the algorithms are dynamically adjusted to avoid a large test suite size caused by the premature convergence. The research conducted the parameter-adjusting experiments for quantum particle swarm optimization(Fig. 2) and presented the recommended parameter setting.

(a)iteration

(b)population size

(c) initial temperature

(d) population reinitialization threshold

Fig. 2 Parameter-adjusting experiment diagram

Finally, with the ablation of various experimental benchmarks and different contrastive algorithms(as is shown in Table. 2), a lot of contrastive experiments(as is shown in Table. 3) and Wilcoxon signed rank test for experiments results(as is shown in Table. 4), the results of the experiments clearly show that the algorithm proposed in the article can generate more competitive results than most commonly used methods in terms of efficiency and time performance. And the results have huge differences. The proposed method successfully builds test suites where strength is up to 15 and totally reports 13 new best test suite size records.

Table. 2 Results of ablation experiments

Table 3 Results of contrastive experiments

Table. 4 Statistical testing analysis of ablation experiment results

IEEE Transactions on Software Engineering(IEEE TSE) is an international top journal in software engineering which is graded as an “international top journal” by China Computer Federation (CCF) and a Class A journal where CCF encourages Chinese scholars to publish their articles. The journal is listed among the journals in the first section of SCI indexed journals of Chinese Academy of Sciences and a top journal in the software engineering. Launched in 1975, IEEE TSE has had a history of 48 years and enjoys wide citations and recognition. The article is the first research result that is published by IMU in IEE TSE, which marks that the research of IMU on the software and theories of computer science has entered a new phase.

Prof. Zhou Jiantao with the College of Computer Science(College of Software) of IMU is the corresponding author of the article. Guo Xu is the first author of the article who graduated as a PhD from the College of Computer of Science of IMU and began to work at IMU in June 2023. IMU is the institution the first author is with.

URL of the article: https://ieeexplore.ieee.org/document/10224655/authors#authors

Prof. Zhou Jiantao(Left) and Guo Xu(Right)