Also note that P is the submitted set of parameters, V is the set of values of the parameters, and t is the strength. As we have just pointed out, TTR 1.1 follows the same general 3 steps as we have in TTR 1.0. PICT can be regarded as one baseline tool where other approaches have been done based on it (PictMaster 2017).
The details of the input space model (ISM) construction, such as factor identification and value assignment, are included. We compared the faults detected by CT with those detected by the in-house testing teams using other methods, and the results suggest that despite some challenges, CT is an effective technique to detect real faults, especially multi-factor faults, of software systems in industrial settings. Observations and lessons learned are provided to further improve the fault detection effectiveness and overcome various challenges. We performed two controlled experiments addressing cost-efficiency and only cost. Considering both experiments, we performed 3,200 executions related to 8 solutions.
iTree: Efficiently Discovering High-Coverage Configurations Using Interaction Trees
Combinatorial testing can help detect problems like this early in the testing life cycle. The key insight underlying this method is that not every parameter contributes to every failure and most failures are triggered by a single parameter value or interactions between a relatively small number of parameters. To detect interaction failures, software developers often use “pairwise testing”, in which all possible pairs of parameter values are covered by at least one test. Its effectiveness is based on the observation that software failures often involve interactions between parameters. For example, a router may be observed to fail only for a particular protocol when packet volume exceeds a certain rate, a 2-way interaction between protocol type and packet rate. Note that the failure will only be triggered when both pressure 300 are true.
In Consumer Electronics (ISCE 2014), The 18th IEEE International Symposium on (pp. 1-2). TTR was implemented in Java and C (TTR 1.2) and we developed three versions of our algorithm. In this paper, we focused on the description of versions 1.1 and 1.2 since version 1.0 was detailed elsewhere (Balera and Santiago Júnior 2015). It is an adaptation of IPOG where constraint handling is provided via a SAT solver. The greatest contribution are three optimizations that seek to reduce the number of calls of the SAT solver. As IPOG-C is based on IPOG, it accomplishes exhaustive comparisons in the horizontal growth which may lead to a longer execution.
N-wise testing
This paper reports on a case study done for evaluating and revisiting a recently introduced combinatorial testing methodology used for web application security purposes. It further reports on undertaken practical experiments thus strengthening the applicability of combinatorial testing to web application security testing. Applied combinatorial testing to industrial control systems, using mixed-strength covering arrays, “resulting in requiring fewer tests for higher strength coverage”. IPO-TConfig is an implementation of IPO in the TConfig tool (Williams 2000). The TConfig tool can generate test cases based on strengths varying from 2 to 6.
Abstract—Modern passenger cars have a comprehensive embedded distributed system with a huge number of bus devices interlinked in several communication networks. The number of (distributed) features and hence the risk of undesired feature interaction within this distributed system rises significantly. Such distributed automotive features pose a huge challenge in terms of efficient testing.
How does combinatorial testing perform in the real world: an empirical study
Note that this is not the exact number of tests produced; the test set size is proportional to this value, i.e., the number of tests grows exponentially with the number of values, but only logarithmically with the number of parameters. This size is a characteristic of covering arrays, and holds for all covering array generating tools, not just ACTS. For the tester, this means that it is best to keep the number of values per parameter under about 10, but it is not a problem to have hundreds of parameters. Combinatorial methods can be applied to all types of software, but are especially effective where interactions between parameters are significant. The primary industry applications for ACTS are in database and e-commerce, aerospace, finance, telecommunications, industrial controls, and video game software, but we have users in probably every industry. In general, we can say that IPOG-F presented the best performance compared with TTR 1.2, because IPOG-F was better for all strengths, as well as lower and medium strengths.
As the order in which the parameters are presented to the algorithms alters the number of test cases generated, as previously stated, the order in which the t-tuples are evaluated can also generate a certain difference in the final result. These results are interesting because they suggest that, while pairwise testing is not sufficient, the what is combinatorial testing degree of interaction involved in failures is relatively low. Testing all 4-way to 6-way combinations may therefore provide reasonably high assurance. As with most issues in software, however, the situation is not that simple. Most parameters are continuous variables which have possible values in a very large range (+/- 232 or more).
Therefore, considering the metrics we defined in this work and based on both controlled experiments, TTR 1.2 is a better option if we need to consider higher strengths (5, 6). For lower strengths, other solutions, like IPOG-F, may be better alternatives. The general description of both evaluations (cost-efficiency, cost) of this second study is basically the same as shown in Section 4. Algorithms/tools were subjected to each one of the 80 test instances, one at a time, and the outcome was recorded. Cost is the number of generated test cases, and efficiency was obtained via instrumentation of the source code with the same computer previously mentioned.
- Several approaches are based on IPO such as IPOG, IPOG-D (Lei et al. 2007), IPOG-F, IPOG-F2 (Forbes et al. 2008), IPOG-C (Yu et al. 2013), IPO-TConfig (Williams 2000), ACTS (where IPOG, IPOG-D, IPOG-F, IPOG-F2 are implemented)(Yu et al. 2013), and CitLab (Cavalgna et al. 2013).
- Last but not least, we claim that attack pattern-based combinatorial testing with constraints can be an alternative method for web application security testing, especially when we compare our method to other test generation techniques like fuzz testing.
- Recent empirical studies show that meta-heurisitic and greedy algorithms have similar performance (Petke et al. 2015).
- PICT can be regarded as one baseline tool where other approaches have been done based on it (PictMaster 2017).
- Develops a method of applying combinatorial testing for use with SQL database query programs.
In addition, we need to determine the correct result that should be expected from the system under test for each set of test inputs. But these challenges are common to all types of software testing, and a variety of good techniques have been developed for dealing with them. If branch coverage is not close to 100%, then (1) input parameter values can be changed to improve the test set, or (2) a higher strength (higher value of t) covering array can be used. The question below on combinatorial coverage explains why this heuristic is important.