Matrix Dimensions Randomization

A dimension does not have to list its values. It can describe how to draw them instead: a few numbers from a range, a few entries from a pool, or a few strings built from a character set. This page follows Matrix Cases Exclusion and uses the Matrix Dimensions Randomization test of the Advanced Configuration Parameters group.

What stays fixed is the number of values each dimension contributes. The matrix therefore produces the same number of cases on every configuration, which is what lets the controller plan a randomized test the way it plans any other, while the values themselves change from one draw to the next.

Select the Matrix Dimensions Randomization test and set its Maze Executable Folder:

The panel reports Total Cases: 48. Four of the six column headers, seed, height, width_ratio and illegal_arg, carry a dice icon: these dimensions drew their values when the panel was built. Here seed drew 100 and 2024 from its pool, height drew 15 and 9, width_ratio drew two floating-point numbers and illegal_arg drew GM and G. Another panel would show other values, but always two per dimension.

Defining random dimensions

Source directory:

The random dimensions are declared with the Dimension class methods, next to two ordinary dimensions that list their values:

matrix = OrcraftConfigField.Matrix(
	OrcraftConfigField.Dimension(label="command", values=("generate", "solve")),
	OrcraftConfigField.Dimension(label="algorithm", values=(None, "BFS")),
	OrcraftConfigField.Dimension.random_choice(label="seed", values=(1, 7, 42, 100, 2024), size=2),
	OrcraftConfigField.Dimension.random(label="height", min_value=5, max_value=15, size=2),
	OrcraftConfigField.Dimension.random(label="width_ratio", min_value=1.5, max_value=2.0, size=2),
	OrcraftConfigField.Dimension.random_string(label="illegal_arg", min_length=1, max_length=2, size=2, charset=" MG"),

	random_order=True,

	exclude=[
		lambda case: case.params["command"] == "generate" and case.params["algorithm"] is not None,
	],
)

Each constructor describes a draw and the size of it:

• random_choice takes size values from the values pool, without repeating one.
• random takes size numbers from the inclusive range between min_value and max_value. Integer bounds give integers, as for height; floating-point bounds give floating-point numbers, as for width_ratio.
• random_string takes size strings whose length lies between min_length and max_length, built from charset. A single length can be given instead of the two bounds, and without a charset letters and digits are used.

The declaration fixes the case count. Two commands, two algorithms and four dimensions of two values give 2 × 2 × 2 × 2 × 2 × 2 = 64 combinations, and the exclusion rule removes the sixteen generate cases that name an algorithm, leaving 48. The rule refers only to the two listed dimensions, so it removes the same sixteen cases whatever the random dimensions draw and the count stays 48. A rule that looked at a random value could remove a different number of cases on each draw, and the panel warns about that combination. The random_order option is covered below.

Using the drawn values

The drawn values reach _run_case through case.params like any other dimension value. The test derives the maze width from the height and the ratio, and appends the random string to the command:

def _run_case(self, case: OrcraftTestCase):

	maze_app = MazeApp(
		self.config.maze_executable_folder,
		(case.params["command"] + case.params["illegal_arg"]).strip(),
		width=round(case.params["height"] * case.params["width_ratio"]),
		height=case.params["height"],
		seed=case.params["seed"],
		algorithm=case.params["algorithm"],
	)

	maze_app.run()

	return case.result

A command such as generateGM is not one the Maze application knows, and its complaint appears in the case log. The case still passes, because this test only exercises the values and checks nothing. Turning an application's answer into a case status is the subject of the Statuses and Results group later in the tutorial.

The random configuration

Click the seed column header. The Dimension Filter dialog introduced in Filtering Matrix Dimensions opens, but for a random dimension its lower half, Random Configuration, is where the dimension is defined:

Mode reads Random Choice, Values holds the pool declared in the source and Size the number of values to draw. The list above shows the two values this draw produced, both checked. Select All and Clear All are disabled and the Add value field is absent: the list is the result of the draw, not a set of choices to narrow.

The height dimension shows the numeric form of the same dialog:

Random Number exposes the Min and Max bounds and the Size. The illegal_arg dimension shows the string form:

Random String exposes Min Length, Max Length, Size and Charset. The charset declared in the source is " MG" with a leading space; the strings are stripped of it in _run_case, which is how a draw can also produce a plain generate with no suffix at all.

Drawing again

The specification is the configuration. The values it happened to draw are not, so a run does not draw new values by itself: the values on the panel are the values the run uses, and a second run of the same panel uses them again. To draw again, open the dimension's dialog and click Apply or OK, even with nothing changed. The dimension is drawn afresh, the case table is rebuilt around the new values and the plan is marked as changed until Update Plan commits it.

Changing the specification

Every field of the Random Configuration group can be edited for this configuration. Set the seed dimension's Size to 3 and click OK:

Three seeds instead of two make Total Cases: 72, and both the seed header and the counter carry the 🔸 badge because the panel no longer matches the committed plan. The Mode itself can be changed too, so a dimension declared with a pool can be re-specified as a numeric range from the controller, and Disabled turns the dimension back into a plain list of the values currently drawn. Reset to Defaults inside the dialog restores the declared specification, and the panel's own Reset To Defaults restores every dimension at once.

Run order

The Run Order control above the case table decides the order in which the selected cases run. In Order runs them by case number. Random (New) shuffles them with a fresh seed, and the Order column shows the position each case will take. Once a shuffled run exists, Random (Last) repeats its order. The matrix declares random_order=True to ask for a shuffled order by default; the value is part of the matrix options, together with the seed options that make a shuffled run reproducible.

Running the drawn cases

With the declared specification in place, click Update Plan, then Run and confirm with Yes. The test reports 48 planned, done and passed cases, and the log's result table records the drawn value of every dimension for every case:

The log is what makes a random run reproducible after the fact. A case that fails in a soak run can be re-run by hand with the exact values the table shows, and the full floating-point value of width_ratio appears here even where the panel had to truncate it.

Inspecting the run

Click Details for the completed run. The configuration panel on the left has a Random Dimensions table that records the specification of each random dimension, with its default, planned and run values side by side: the mode, the pool or range, the length bounds, the size and the charset. The listed dimensions keep their own table below it:

The dialog records the specification because that is what was configured. The values the run drew are in the case table further down and in the log, so both halves of a random run, the rule and the outcome, can be inspected later.

The values on this page are drawn by the framework. A dimension can also obtain its values from a function the test supplies, which is what Matrix Lambda Dimensions covers next.