Structure and Lifecycle

The tests so far did their work inside _run_case. A real test is usually built from smaller parts: steps that each do one thing, and flows that run steps in order. Around the cases, optional lifecycle hooks prepare and release what the cases share. The Structure and Lifecycle group shows both, one test at a time: Steps and Flows composes a test body, and the three setup and teardown tests each add one lifecycle scope to the previous one.

Select each of the four tests, set its Maze Executable Folder, and click Update Plan. The field is a package folder in this group, a type that can upload the folder to a remote endpoint; on the local machine it behaves like the plain folder used so far.

Steps and flows

Source directory:

orcraft-test-suite-tutorial/test_group_structure_and_lifecycle/test_steps_and_flows/

The test folder holds the test file and two subfolders, flows and steps. The case body runs two flows in turn. The first generates a maze, saves it to a file, loads it back and solves it. The second does the same but corrupts the saved file first, and expects the solver to find no path:

def _run_case(self, case: OrcraftTestCase) -> OrcraftTestResult:
	case.result = TestStepsAndFlowsResult()

	# Flow: Generate/Save -> Load/Solve
	flow_generate_save = FlowGenerateSaveLoadSolve(
		FlowGenerateSaveLoadSolveConfig(
			maze_executable_folder=self.config.maze_executable_folder,
			width=case.params["width"],
			height=case.params["height"],
		)
	)
	flow_generate_save_result = flow_generate_save.run()

	if self.handle_errors(flow_generate_save_result, case.result):
		return case.result

	case.result.maze_area = flow_generate_save_result.maze_area
	case.result.path_length = flow_generate_save_result.path_length
	case.result.area_path_ratio = flow_generate_save_result.area_path_ratio
	case.result.maze_file_path = flow_generate_save_result.maze_file_path

	# Flow: Generate/Save -> Corrupt Saved File -> Load/Solve
	flow_generate_save_corrupt = FlowGenerateSaveCorruptLoadSolve(
		FlowGenerateSaveCorruptLoadSolveConfig(
			maze_executable_folder=self.config.maze_executable_folder,
			width=case.params["width"],
			height=case.params["height"],
		)
	)

	flow_generate_save_corrupt_result = flow_generate_save_corrupt.run()

	if self.handle_errors(flow_generate_save_corrupt_result, case.result):
		return case.result

	return case.result

A flow is built from its config class and run with run(), like a test. After each flow the case calls handle_errors, which copies the flow's status and descriptions onto the case result and returns True when the flow failed, so the case stops there. The values the first flow measured are copied into the case's own result fields.

A flow

The flow declares its config and result classes and names its result class as result_type. Its body runs steps with run_step, passing each step's config values as keyword arguments, and checks each step result the same way the case checks the flow:

class FlowGenerateSaveLoadSolve(OrcraftTestFlow):
	result_type = FlowGenerateSaveLoadSolveResult

	def __init__(self, config: FlowGenerateSaveLoadSolveConfig) -> None:
		super().__init__(config)

	def _run_flow(self) -> None:
		step_1_result = self.run_step(
			StepDeleteExistingMazeFile,
			width=self.config.width,
			height=self.config.height,
		)

		if self.handle_errors(step_1_result, self.result):
			return

		step_2_result = self.run_step(
			StepGenerateSave,
			maze_executable_folder=self.config.maze_executable_folder,
			maze_file=step_1_result.maze_file,
			width=self.config.width,
			height=self.config.height,
		)

		if self.handle_errors(step_2_result, self.result):
			return

		self.result.maze_file_path = step_2_result.maze_file_path
		self.result.maze_area = step_2_result.maze_area

		step_3_result = self.run_step(
			StepLoadSolve,
			maze_executable_folder=self.config.maze_executable_folder,
			maze_file=step_2_result.maze_file_path,
		)

		if self.handle_errors(step_3_result, self.result):
			return

		self.result.path_length = step_3_result.path_length

		if step_3_result.path_found and step_3_result.path_length > 0:
			self.result.area_path_ratio = round(step_2_result.maze_area / step_3_result.path_length, 2)
			self.checkpoint(
				step_3_result.path_length <= step_2_result.maze_area,
				f"Path length ({step_3_result.path_length}) is within maze area ({step_2_result.maze_area}).",
				f"Path length ({step_3_result.path_length}) exceeds maze area ({step_2_result.maze_area}) — unexpected.",
			)

		self.checkpoint(
			step_3_result.path_found,
			f"Path found: {step_3_result.path_length} step(s).",
			"No path found in maze output.",
		)

Results travel forward through the flow: the file path the first step chose is handed to the second, and the path the second wrote is handed to the third. The flow's own checkpoints judge the combined outcome.

A step

A step is the smallest unit, and the unit that runs on an endpoint. It has a config class, a result class and a _run_step body:

class StepLoadSolveConfig(OrcraftTestStepConfig):
	name = OrcraftConfigField.Text(default="Load / Solve")
	maze_executable_folder = OrcraftConfigField.PackageFolder(default="", label="Maze Executable Folder")
	maze_file = OrcraftConfigField.Text(default="", label="Maze File")
	log_orientation = OrcraftConfigField.Format(default=OrcraftConfigField.LogTableOrientation.VERTICAL)


class StepLoadSolveResult(OrcraftTestStepResult):
	path_found = OrcraftResultField.Boolean(default=False, label="Path Found")
	path_length = OrcraftResultField.Numeric(default=None, label="Path Length")
	log_orientation = OrcraftResultField.Format(default=OrcraftResultField.LogTableOrientation.VERTICAL)


class StepLoadSolve(OrcraftTestStep):
	def __init__(self, config: StepLoadSolveConfig | None = None) -> None:
		super().__init__(config or StepLoadSolveConfig())

	def _run_step(self) -> OrcraftTestStepResult:
		self.result = StepLoadSolveResult()

		maze_file = normalize_long_path(self.config.maze_file)

		self.checkpoint(
			bool(maze_file),
			"Maze file path is set.",
			"Maze file path is empty.",
		)

		if not maze_file:
			return self.result

		self.checkpoint(
			os.path.exists(maze_file),
			f"Maze file exists: {maze_file}",
			f"Maze file does not exist: {maze_file}",
		)

		if not os.path.exists(maze_file):
			return self.result

		result = MazeApp(
			normalize_long_path(self.config.maze_executable_folder),
			"solve",
			maze_file=maze_file,
		).run()

		self.result.path_found = result.solved

		if result.solved:
			self.result.path_length = result.steps

		return self.result

The same three classes, config, component and result, appear at every level of the framework. A step's log_orientation field only changes how its result table is printed: vertically, one parameter per line, which reads better for the few values a step reports.

The composed run

Run Steps and Flows. When it finishes, search the log for Corrupted maze has no solution as expected:

The log nests the same way the code does. The Load / Solve step prints its own result table with Path Found false, and the flow's checkpoint right after it confirms that this is the expected outcome for a corrupted maze. The Execution Status frame at the bottom right lists the stages of the run as they nest: the test, its cycle, the case, the case cycle, the flow and the step, the last one marked with the endpoint it ran on.

Click Details for the run:

Every flow and every step is a row of its own beneath the case cycle, with its status, its timing and, for a step, the endpoint it ran on. A failure anywhere in this tree is reported at the row that produced it and surfaced at every row above it.

Setup and teardown

Source directory:

orcraft-test-suite-tutorial/test_group_structure_and_lifecycle/test_setup_and_teardown/

Setup and Teardown generates its maze files once, before any case, and deletes them once, after the last case. Each of its four cases loads and solves one of the files, chosen by the case's maze_index. The two hooks are methods on the test:

def _setup(self) -> None:
	workspace_folder = self._workspace_folder()
	maze_count = int(self.config.maze_count)

	Logger.log(
		f"Setup: generating {maze_count} maze files in {workspace_folder}",
		color=Logger.Color.CYAN,
	)

	flow = FlowGenerateMazeFiles(
		FlowGenerateMazeFilesConfig(
			maze_executable_folder=self.config.maze_executable_folder,
			workspace_folder=workspace_folder,
			count=maze_count,
		)
	)

	# Raising from _setup aborts the test: no cases run and _teardown is skipped.
	if self.handle_errors(flow.run(), self.result):
		raise RuntimeError(
			self.result.error_description
			or self.result.failure_description
			or "Setup failed to generate the maze files."
		)

def _teardown(self) -> None:
	workspace_folder = self._workspace_folder()

	Logger.log(
		f"Teardown: deleting maze files in {workspace_folder}",
		color=Logger.Color.CYAN,
	)

	# Teardown does not raise — cleanup problems should not mask case results.
	flow = FlowDeleteMazeFiles(
		FlowDeleteMazeFilesConfig(
			workspace_folder=workspace_folder,
		)
	)
	flow.run()

A hook can run flows and steps like a case can. The two comments state the rule that matters: a setup that raises aborts the test, because the cases would run without what they need, while this teardown swallows its problems so that a cleanup failure never hides the cases' results. A teardown that raised instead would mark the test as an error while the cases kept their results. Run the test and open its Details:

Setup and Teardown are rows of the results tree in their own right, each with the flow and step it ran, placed before and after Cycle 1. Their timing is separate from the cases: generating four mazes took most of the run, and the cases themselves were quick.

Cycle setup and teardown

Source directory:

orcraft-test-suite-tutorial/test_group_structure_and_lifecycle/test_cycle_setup_and_teardown/

Cycle Setup and Teardown subclasses the previous test and keeps its hooks. It sets Cycles to two and adds the per-cycle pair, which receives the cycle index:

def _test_cycle_setup(self, test_cycle_index: int) -> None:
	...
	# Raising here fails only this cycle; the next cycle still runs.
	if self.handle_errors(flow.run(), self.result):
		raise RuntimeError(...)

def _test_cycle_teardown(self, test_cycle_index: int) -> None:
	...
	# Teardown does not raise — cleanup problems should not mask cycle results.
	FlowClearCycleMazes(
		FlowClearCycleMazesConfig(
			workspace_folder=workspace_folder,
		)
	).run()

The cycle setup regenerates the maze files, so every cycle starts from fresh inputs, and the cycle teardown clears them again, leaving the folder itself to the test-level teardown. Run the test and open its Details:

Each Cycle now opens with a Test Cycle Setup row and closes with a Test Cycle Teardown row, inside the test-level Setup and Teardown that still bracket the whole run. A raise in a cycle setup fails that cycle only.

Case setup and teardown

Source directory:

orcraft-test-suite-tutorial/test_group_structure_and_lifecycle/test_case_setup_and_teardown/

Case Setup and Teardown subclasses the cycle test in turn, sets Case Cycles to two, and adds the innermost pair, which receives the case and the cycle index:

def _case_setup(self, case: OrcraftTestCase, test_cycle_index: int) -> None:
	maze_index = case.params["maze_index"]
	...
	# Raising from _case_setup fails this case only: its case cycles are
	# skipped and the matching _case_teardown does not run.
	flow_result = flow.run()
	if self.handle_errors(flow_result, case.result):
		raise RuntimeError(...)

def _case_teardown(self, case: OrcraftTestCase, test_cycle_index: int) -> None:
	maze_index = case.params["maze_index"]
	...
	# Teardown does not raise — cleanup problems should not mask case results.
	FlowReleaseCaseMaze(
		FlowReleaseCaseMazeConfig(
			workspace_folder=self._workspace_folder(),
			maze_index=maze_index,
		)
	).run()

The case setup claims the one maze file the case owns and the case teardown releases it. Because the case hooks bracket the case cycles rather than each cycle, a case that repeats its work twice claims its file once. Run the test:

With two cycles of three cases the test plans 12 cases, and the whole group is now green. Open the run's Details:

All the scopes are visible in one tree. Reading from the outside in: Setup, then Cycle 1 with its Test Cycle Setup, then Case 1 with its Case Setup, its two Case Cycle rows and its Case Teardown, and so on outward again. Each hook row holds the flow and step it ran, so a failure in a hook is placed as precisely as a failure in a case.

The hooks and the order they run in are listed in the framework reference, together with what happens when a hook raises at each scope.

Every step on this page ran on the local machine. The next group, Endpoints, sends steps to other machines, starting with Endpoints Management.