Orcraft Matrix

A test does not run once. It runs once per case, and cases come from the matrix - a set of named dimensions whose values are combined. The matrix is configuration rather than code: the same test body serves one combination or four hundred, and which combinations run is decided before the run starts rather than inside the case.

Matrix definition

Cases come from the test matrix: a set of named dimensions whose values are combined.

matrix = OrcraftConfigField.Matrix(
	OrcraftConfigField.Dimension("payload_size", (64, 256)),
	OrcraftConfigField.Dimension("retries", (1, 3)),
)

Two dimensions of two values produce four cases - every combination - each reaching _run_case with its own case.params. Add a third dimension of three values and the same test body now covers twelve cases without a line changing.

The controller shows the resulting case list before a run and lets you select, deselect and filter it, so the plan is explicit rather than discovered mid-execution. The tutorial follows that through in Matrix and cases and Filtering matrix dimensions.

Dimensions

Dimension values do not have to be listed by hand. Five constructors cover the common cases:

Constructor	Produces
`Dimension(label, values)`	The values given, in order.
`Dimension.random(label=, min_value=, max_value=, size=)`	`size` numbers sampled from the inclusive range.
`Dimension.random_choice(label=, values=, size=)`	`size` values sampled from `values` without replacement.
`Dimension.random_string(label=, size=, length=)`	`size` random strings of a fixed `length`, or within `min_length` and `max_length`, optionally from a given `charset`.
`Dimension.lambda_choice(label=, factory=)`	Values produced by calling `factory()` when the run is configured.

matrix = OrcraftConfigField.Matrix(
	OrcraftConfigField.Dimension(label="mode", values=("fast", "thorough")),
	OrcraftConfigField.Dimension.random_choice(label="seed", values=(1, 7, 42, 100, 2024), size=2),
	OrcraftConfigField.Dimension.random(label="payload_size", min_value=64, max_value=1024, size=2),
	OrcraftConfigField.Dimension.random_string(label="label", min_length=1, max_length=2, size=2, charset="AB "),
	OrcraftConfigField.Dimension.lambda_choice(label="version", factory=fetch_release_versions),
)

The generated variants keep the case count fixed while the values vary, which is what makes a randomized matrix usable in a plan: the controller can still show how many cases will run, and a soak run covers new ground on every cycle without a new test.

lambda_choice is for values that only exist at configuration time - the releases currently published, the devices currently attached. The factory runs when the run is configured, not when the test file is imported.

Exclusion rules

A full cross product usually contains combinations that make no sense. exclude removes them before they ever become cases, as either a dict of values to match or a predicate over the case:

matrix = OrcraftConfigField.Matrix(
	OrcraftConfigField.Dimension("operation", ("create", "read", "update", "delete")),
	OrcraftConfigField.Dimension("strategy", (None, "sequential", "parallel")),
	OrcraftConfigField.Dimension("use_cache", (False, True)),

	exclude=[
		{"operation": "create", "use_cache": True},
		{"operation": "read", "use_cache": False},
		lambda case: case.params["operation"] == "delete" and case.params["strategy"] is not None,
	],
)

A dict rule excludes every case matching all of its keys; a lambda excludes every case it returns True for. Excluded combinations are not skipped at run time - they are never planned, so the case count the controller shows is the case count that runs.

Matrix options

Option	Meaning
`exclude`	Combinations that must not become cases.
`random_order`	Shuffles the case order rather than running the combinations in sequence.
`random_seed`	Fixes the seed so a randomized matrix is reproducible.
`random_seed_per_cycle`	Draws a fresh seed for each test cycle instead of reusing one.
`name_prefix`	Prefix for generated case names.
`result_fields`	Which result fields are shown per case in the case table.
`selected_cases`	The subset of cases selected to run by default.

random_seed and random_seed_per_cycle are the pair that decides whether a randomized run is an experiment or a regression. A fixed seed reproduces the exact same values every time, which is what you want once a random run has found something; a per-cycle seed covers new values on every cycle, which is what you want while looking.

Orcraft Test Case

Each combination the matrix produces arrives at _run_case as an OrcraftTestCase, carrying that case's parameter values and the result it will report:

def _run_case(self, case: OrcraftTestCase) -> OrcraftTestResult:
	case.result = MyTestResult()

	measured = measure(
		payload_size=case.params["payload_size"],
		retries=case.params["retries"],
	)
	...
	return case.result

Member	Meaning
`case.params`	Mapping of dimension name to this case's value. `case["width"]` is shorthand for `case.params["width"]`.
`case.result`	The result this case reports. Assign your own `OrcraftTestResult` subclass to it.
`case.config`	The case's own config, carrying `number_of_cycles` for this case - see Cycles.

Cases are generated, never written. A test with no matrix still runs exactly one case, with case.params simply empty - which is what keeps the simplest possible test and a large parameterized one on the same code path.

How many times each case runs is a separate question from how many cases there are; both counters live on the test config and are covered in Cycles.