Matrix Cases Exclusion

A full Cartesian product usually contains combinations that make no sense for the application under test. Exclusion rules remove them in the test source, before they ever become cases. This page follows Filtering Matrix Dimensions, which narrows a matrix from the controller, and uses the Cases Exclusion test of the same Advanced Configuration Parameters group.

The difference between the two is where the decision lives. A dimension filter is made in the run panel for one configuration and can be lifted again. An exclusion rule is part of the test's matrix declaration: every run of the test, from the GUI, the CLI or a CI pipeline, leaves those combinations out.

Select the Cases Exclusion test. It runs the Maze application with a command and a mix of optional arguments, so it needs both the Maze Executable Folder and a Maze Save/Load File for the commands that save or load a maze. Set both, for example to the v3.0.0 folder and its maze.json:

The panel reports Total Cases: 61 and the group header reads Test Cases (61/61). Six dimensions with 4 × 4 × 2 × 2 × 2 × 2 values would produce 256 combinations; the matrix produces 61 because the rest are excluded. Nothing marks that in the panel. Unlike a filtered dimension, whose column header is flagged as changed, an excluded combination was never part of the matrix, so there is nothing to lift and nothing to badge. The execution tree's Plan column shows the same 61.

Defining exclusion rules

Source directory:

The Matrix declaration takes an exclude list next to its dimensions. Each entry is either a dict of values to match or a lambda over the case:

matrix = OrcraftConfigField.Matrix(
	OrcraftConfigField.Dimension("command", ("generate", "solve", "save", "load")),
	OrcraftConfigField.Dimension("algorithm", (None, "BFS", "DFS", "ASTAR")),
	OrcraftConfigField.Dimension("seed", (None, 42)),
	OrcraftConfigField.Dimension("height", (None, 5)),
	OrcraftConfigField.Dimension("width", (None, 10)),
	OrcraftConfigField.Dimension("use_file", (False, True)),

	exclude=[
		{"command": "generate", "algorithm": "BFS"},
		{"command": "generate", "algorithm": "DFS"},
		{"command": "generate", "algorithm": "ASTAR"},
		{"command": "generate", "use_file": True},
		{"command": "load", "use_file": False},
		lambda case: case.params["command"] in ("save", "load") and case.params["algorithm"] is not None,
		lambda case: case.params["command"] == "load" and any(case.params[p] is not None for p in ("seed", "height", "width")),
		lambda case: case.params["command"] == "solve" and case.params["use_file"] and any(case.params[p] is not None for p in ("seed", "height", "width")),
	],
)

A value of None in a dimension means the argument is not passed to the Maze application at all. The run panel shows such a value as an empty cell, the log table as -, and the Details dialog as None.

Dict rules

A dict rule names one value per dimension it cares about and excludes every case that matches all of them. {"command": "generate", "use_file": True} removes every generate case that would also pass a file, whatever its algorithm, seed, height or width. A dict rule cannot express "any value but this one", which is why generate needs three rules to exclude the three real algorithms while leaving None in.

Lambda rules

A lambda receives the candidate case and excludes it by returning True. It can compare several dimensions, test for None, and use in or any over a set of keys. The rule case.params["command"] in ("save", "load") and case.params["algorithm"] is not None replaces six dict rules, one per command and algorithm pair, with a single line. The last rule keeps solve from both loading a maze from a file and asking for a freshly generated one.

Both kinds can be mixed in the same list, and a case is excluded as soon as any rule matches it. What survives is the product of the values kept for each command:

• generate - no algorithm and no file, so only seed, height and width vary: 8 cases.
• solve - every algorithm with generation arguments and no file, plus every algorithm with a file and no generation arguments: 36 cases.
• save - no algorithm, everything else free: 16 cases.
• load - a file and nothing else: 1 case.

That is the 61 the panel reports. Excluded combinations are not skipped at run time. They are never planned, so the case count the controller shows before a run is the case count that runs.

Using the surviving cases

The test body is the same shape as in Matrix and Cases. It receives only the cases that passed the rules, so it does not have to guard against the excluded combinations itself:

def _run_case(self, case: OrcraftTestCase):

	maze_app = MazeApp(
		self.config.maze_executable_folder,
		case.params["command"],
		width=case.params["width"],
		height=case.params["height"],
		seed=case.params["seed"],
		algorithm=case.params["algorithm"],
		maze_file=self.config.maze_file if case.params["use_file"] else None,
	)

	maze_app.run()

	return case.result

The use_file dimension decides whether the configured Maze Save/Load File is passed along. Because the rules already removed load without a file and generate with one, the body can pass the file whenever the dimension says so.

Running the surviving cases

Click Update Plan to commit the two configured paths, then Run and confirm with Yes. The test row reports 61 planned, done and passed cases, and the log's result table lists every surviving combination in matrix order:

The order of the table follows the dimensions as declared: all generate cases first, then solve, then save, and the single load case last. Within a command the later dimensions change fastest, exactly as they would without exclusion, only with the removed rows missing.

Inspecting the run

Click Details for the completed run. The List Dimensions table on the left still lists every value each dimension declares, including the ones no surviving case uses together, because the rules act on combinations rather than on values. The case table below it and the results tree on the right hold the 61 cases that exist:

An exclusion rule therefore changes what the matrix is, not how it is shown. Filters and case selection in the run panel still apply on top of it, so a reader can narrow the 61 cases further for one configuration in the same way as on any other matrix test.

All the dimensions on this page list their values by hand. The next page, Matrix Dimensions Randomization, lets a dimension draw its values at random and shows how the controller keeps such a matrix plannable.