Matrix and Cases

Matrix and Cases generates one test case for every combination of values declared by the test's matrix. It follows Lambda Configuration Parameters, which introduces callback-driven configuration choices in the same tutorial suite. Select the test to see the generated cases below its regular configuration fields:

The summary reports Total Cases: 24, and the table shows how each case was assembled. The algorithm, seed, height and width columns come directly from the matrix dimensions. Case is the generated case number, while Order is the position in which that case will run. With Run Order set to In Order, those values are the same (the matrix options can shuffle it).

Each check mark in Select includes that case in the run. Clearing one omits only that combination; it does not change the matrix definition. Case Cycles controls how many times each selected case runs, independently of the test-level Cycles value - see Cycles.

Defining the matrix

Source directory:

Declare a Matrix field on the test config and add one Dimension for each case parameter. The first argument is the key used in the generated case; the second is the set of values that dimension can take:

matrix = OrcraftConfigField.Matrix(
	OrcraftConfigField.Dimension("algorithm", ("BFS", "DFS", "ASTAR")),
	OrcraftConfigField.Dimension("seed", (42, 100)),
	OrcraftConfigField.Dimension("height", (5, 10)),
	OrcraftConfigField.Dimension("width", (15, 20)),
)

Orcraft builds the Cartesian product of those values. This matrix therefore creates 3 × 2 × 2 × 2 = 24 cases. In the default order, the last dimension changes fastest: width alternates first, followed by height, seed and then algorithm, which is the sequence visible in the table.

Using matrix values

Orcraft creates an OrcraftTestCase for each row in the Cartesian product, stores that row's dimension choices in the case's params mapping, and passes the case to the test's _run_case method.

For example, the first row in the screenshot produces a mapping equivalent to:

{
	"algorithm": "BFS",
	"seed": 42,
	"height": 5,
	"width": 15,
}

The complete test class receives that generated case and uses its values to configure the Maze application:

class TestMatrixAndCases(OrcraftTest):
	def __init__(self, config: TestMatrixAndCasesConfig | None = None) -> None:
		super().__init__(config or TestMatrixAndCasesConfig())

	def _run_case(self, case: OrcraftTestCase):

		maze_app = MazeApp(
			self.config.maze_executable_folder,
			"solve",
			width=case.params["width"],
			height=case.params["height"],
			seed=case.params["seed"],
			start_row=0,
			start_col=0,
			end_row=case.params["height"] - 1,
			end_col=case.params["width"] - 1,
			algorithm=case.params["algorithm"],
		)

		maze_app.run()

		return case.result

The case: OrcraftTestCase parameter is the framework interface for each generated combination. Every key used with case.params matches a dimension key in the matrix declaration above. Orcraft calls _run_case once for each selected combination, using the configured order and cycle counts.

Running the generated cases

With all 24 cases selected and Case Cycles set to 1, click Run, then click Yes to confirm. Orcraft executes each matrix combination once. The test row reports 24 planned, completed, and passed items, while the log presents one result row for every generated case:

The result table preserves the matrix context for every execution. Its algorithm, seed, height, and width columns identify the exact combination used by each case, so a failure can be traced to its inputs without reconstructing the matrix order.

Inspecting case results

Click Details for the completed test run to inspect the full execution hierarchy. The left panel compares default, planned, and run configuration, including the matrix dimensions and the cases included in the plan. The results tree on the right contains one node for each case and one Case Cycle 1 child beneath it:

The summary distinguishes generated cases from the surrounding result nodes. In this run, all 24 planned cases completed and passed. Expanding a case exposes its status, timing, and the cycle execution associated with that matrix combination.

Repeating each case

Case Cycles controls repetition within each generated case. Change it from 1 to 2 while all 24 cases are selected. Total Cases immediately changes from 24 to 48, reflecting 24 selected cases multiplied by two case cycles:

This setting is independent of the test-level Cycles value. Test cycles repeat the complete test, while case cycles repeat each selected case within a test cycle; the two counters and their execution order are described in Cycles.

Planning and running case cycles

Click Update Plan to commit the new case-cycle count. The Case Cycles value and Total Cases: 48 indicator turn blue. The previous run is marked as a plan mismatch because it executed each case only once:

Click Run and confirm the run. The completed test reports 48 planned, completed, and passed executions. The log records each repetition independently; the example below shows the configuration, Maze output, and passing result for Case Cycle 2:

Click Details for the completed test run. Every generated case contains two children, Case Cycle 1 and Case Cycle 2, with independent status and timing:

The hierarchy makes the multiplication visible: the matrix defines the cases, case selection determines which combinations participate, and Case Cycles determines how many times each selected combination runs.

Removing a run

This removal is not required for the Matrix and Cases workflow; it is included to demonstrate how Orcraft removes an individual run instance. The execution tree retains a separate row for every run. Before returning the configuration to one case cycle, remove the last two-cycle run used in the preceding example. Right-click that run and click Remove:

After removal, the earlier one-cycle run remains and is marked with Plan Mismatch because the current configuration still plans two case cycles and 48 executions. Orcraft excludes plan-mismatched runs from the current plan's aggregate statistics. Because the removed two-cycle run was the only run that matched the plan, the test now shows Done: 0, Pass: 0, Fail: 0, Progress: 0%, and Pass %: 0%, even though the retained one-cycle run completed successfully. The Matrix and Cases test and its 48-execution configuration are unchanged:

Remove deletes only the selected run instance and its recorded results.

Restoring one case cycle

Set Case Cycles from 2 back to 1. This reduces Total Cases from 48 to 24 without changing the selected cases or the Maze Executable Folder. Click Update Plan to commit the one-cycle configuration:

Selecting a subset of cases

Clearing the case selection

The matrix continues to define 24 possible combinations even when only a subset is required for a run. Click the Select column header to clear the current selection. The table remains populated, but Total Cases changes to 0 and the configuration is marked as different from the current plan:

Choosing individual cases

Select the individual rows to include in the next run. In this example, cases 3 and 6 are selected, reducing the execution scope to two cases while preserving all 24 generated rows:

Updating the case plan

Click Update Plan to make the two-case selection the planned scope. The selection and Total Cases: 2 indicator turn blue. The earlier 24-case run receives a Plan Mismatch badge because its executed case set no longer matches the new plan. Because that retained run is excluded from the new plan's aggregate statistics, the test shows zero completed and passing cases:

Running the selected cases

Click Run and confirm the run. Orcraft executes only cases 3 and 6, and the completed test reports Plan: 2, Done: 2, Pass: 2, and Pass %: 100%. The log lists both selected matrix combinations, while the older 24-case run remains marked with Plan Mismatch:

Inspecting the selected cases

Click Details for the completed run. In the case table, cases 3 and 6 are the only rows marked as both Planned and Run. The results tree contains those same two cases, each with one passing Case Cycle 1, and the summary reports two planned, completed, and passing executions:

Case selection changes the execution scope, not the matrix definition. Unselected combinations remain available in the table and can be restored without changing the test source.

After choosing which generated cases run, continue to Filtering Matrix Dimensions to change which values a dimension offers, and with them which cases the matrix generates at all.