Result Fields

A status says whether a case was right. A result field says what the case measured. Fields are declared on a result class the way configuration fields are declared on a config class, and every value a case assigns to one is carried into the log, the execution tree and the Details dialog. This page uses the Result Fields test of the Statuses and Results group, which follows Checkpoints.

Select the Result Fields test and set its Maze Executable Folder:

The matrix combines two widths, two heights and three seeds into 12 valid solve cases. Nothing in this test is expected to fail. What it adds is three values per case: the maze area, the length of the path the application found, and the ratio between them.

Declaring result fields

Source directory:

The test declares a result class next to its config class. Each field comes from the OrcraftResultField namespace and carries a default and a label:

class TestResultFieldsResult(OrcraftTestResult):
	maze_area = OrcraftResultField.Numeric(default=None, label="Maze Area")
	path_length = OrcraftResultField.Numeric(default=None, label="Path Length")
	area_path_ratio = OrcraftResultField.Numeric(default=None, label="Area / Path Ratio")

The label is what the log and the controller show. A field without a label stays internal, and the other field types and options are listed in the framework reference.

Assigning values

The case body starts by giving the case an instance of that result class, then hands the case to the suite's Maze helper:

def _run_case(self, case: OrcraftTestCase) -> TestResultFieldsResult:
	case.result = TestResultFieldsResult()

	maze_app = MazeApp(
		self.config.maze_executable_folder,
		"solve",
		width=case.params["width"],
		height=case.params["height"],
		seed=case.params["seed"],
	)

	maze_app.run_and_validate(case)

	return case.result

The helper runs the application, reads the step count from its output and assigns the fields. It also raises a warning when the path is long relative to the maze:

def run_and_validate(self, case: OrcraftTestCase) -> None:
	result = self._execute()
	maze_area = (self.width or 0) * (self.height or 0)
	case.result.maze_area = maze_area
	if result.solved:
		path_length = result.steps
		area_path_ratio = round(maze_area / path_length, 2)
		case.result.path_length = path_length
		case.result.area_path_ratio = area_path_ratio
		if area_path_ratio < 1.8:
			case.result.warning_description = (
				f"Area/path ratio is {area_path_ratio}, which is less than 1.8. "
				"The solution path is long relative to the maze area."
			)

A result field is assigned like any attribute. The warning uses the warning_description seen earlier, so the ratio both appears as a value and, when it is low, flags the case.

Values in the log

Click Update Plan, then Run and confirm. When the run finishes, search the log for Area / Path Ratio and click Next:

Every case ends with a result table, and the three declared fields appear in it under their labels, after the case's dimensions, status and warning. Here a 5 by 15 maze solved in 47 steps gives a ratio of 1.6, low enough for the warning line above the fields.

Clear the search field to return to the end of the log:

The test passes with all 12 cases and reports 6 cases with warnings. The Warnings table lists them with their descriptions, so the low ratios can be found without opening each case.

Values in the Details dialog

Click Details for the run. The declared fields become columns of the results tree, after the columns every result carries:

Each case and case cycle row carries its own Maze Area, Path Length and Area / Path Ratio. The larger mazes at the bottom of the tree have the higher ratios and no warning, and the small ones at the top have the low ratios the warnings pointed at. The columns are part of the dialog's Export to XLSX as well, which is how measured values leave the controller for further analysis.

This completes the Statuses and Results group. The next group, Structure and Lifecycle, starts with Structure and Lifecycle: composing a test from steps and flows, and the setup and teardown hooks that run around a test, a cycle and a case.