Orcraft Setup, Teardown and Exit

Work that has to happen around the cases rather than inside them - launching an application, opening a connection, generating shared inputs - belongs in a setup or teardown method. A test can bracket three scopes, each with a setup and a teardown, and every one of them is optional.

Scope

The scope is a statement about how long the state is valid for. Something expensive that every case can share - an application process, a device connection - belongs in _setup. Something that must be pristine at the start of each pass through the matrix belongs in _test_cycle_setup. Something derived from the case parameters belongs in _case_setup, which is the only one that receives the case.

The framework knows which of these methods a test overrides and shows them in the run plan, so the extra lifecycle work is visible before the run rather than a surprise in the log.

Test scope

def _setup(self) -> None:
	workspace_folder = self._workspace_folder()

	Logger.log(
		f"Setup: preparing {self.config.input_count} input files in {workspace_folder}",
		color=Logger.Color.CYAN,
	)

	flow = MyPrepareInputsFlow(
		MyPrepareInputsFlowConfig(
			app_folder=self.config.app_folder,
			workspace_folder=workspace_folder,
			count=int(self.config.input_count),
		)
	)

	# Raising from _setup aborts the test: no cases run and _teardown is skipped.
	if self.handle_errors(flow.run(), self.result):
		raise RuntimeError(
			self.result.error_description or "Setup failed to prepare the input files."
		)

def _teardown(self) -> None:
	MyCleanupFlow(
		MyCleanupFlowConfig(workspace_folder=self._workspace_folder())
	).run()

Flows and steps called from a setup or teardown inherit the same endpoint and RPC security context as a case body, so setup work can reuse the very flows the cases use. That is worth leaning on: preparing inputs with the same flow that a case exercises keeps one implementation instead of two.

Case scope

_case_setup and _case_teardown receive the case, so they can tailor their work to that case's parameters:

def _case_setup(self, case: OrcraftTestCase, test_cycle_index: int) -> None:
	source = self._input_file_for(case.params["input_index"])
	shutil.copyfile(source, self._working_copy_path(case))

def _case_teardown(self, case: OrcraftTestCase, test_cycle_index: int) -> None:
	os.remove(self._working_copy_path(case))

Both run inside the case, nested beneath its banner in the log, and both wrap the case's cycle loop - so a case that repeats three times gets one setup, three executions and one teardown, not three of each.

Setup and teardown failures

Raising from a setup is how you say the work it was guarding must not proceed. Raising from a teardown says the scope could not be closed cleanly. Either way the blast radius follows the scope of the method, and the run goes on with the next scope of the same kind:

When a setup raises, the matching teardown is skipped, because there is nothing to unwind - a setup that did not complete did not acquire what its teardown would release. The scope's error description names the method, as in "Case setup failed: ..." or "Test cycle teardown failed: ...", followed by the message.

A call to exit() inside a setup or teardown has the same effect as raising from it, with "exit() called by user" as the message; Calling exit() below goes through every place an exit can happen.

The mirror-image guarantee holds for the success path: once a setup completes, its teardown runs even if the work between them raised. Resources acquired in a setup are released whether the cases passed, failed or errored.

Calling exit()

Python's exit() and sys.exit() raise SystemExit, and a plain script ends the moment one is raised. In a suite that would be the wrong outcome: one case giving up must not take the rest of the run with it. So the framework treats an exit exactly like an uncaught exception. The component the call happened in ends with ERROR and the error description exit() called by user, no traceback is logged, and the run goes on with whatever comes next at that level. The exit's own code is discarded: a test run from the command line exits with the framework's error code, the same one an exception produces.

An exit in the case body ends that case. The case cycle it happened in is reported as ERROR, the case is counted as one executed, errored case, its remaining case cycles still run, its _case_teardown still runs, and the next case starts as usual.

An exit inside _run_step ends that step, whether the step runs locally or on an endpoint. On an endpoint the step's result comes back to the runner like any other result, and the endpoint application keeps serving. The flow that ran the step receives the errored step result and, if it hands it on with handle_errors, ends with the same description; the case that ran the flow does the same, and the run continues with the next case. An exit directly inside _run_flow ends the flow the same way.

def _run_flow(self) -> OrcraftTestFlowResult:
	step_result = self.run_step(StepLaunchApp, app_folder=self.config.app_folder)
	# An exit inside the step arrives here as an ERROR result, not as an exception.
	if self.handle_errors(step_result, self.result):
		return self.result
	...

An exit in a setup or teardown ends the scope it guards, exactly as raising from it does: the outcome per method is the Setup and teardown failures table above, with exit() called by user as the message.

The test's own status is the worst of what happened inside it, so a test with one errored case, cycle, setup or teardown reads ERROR even when everything else passed, and its error description is the first one recorded. The counters still tell the full story: the cases that ran and passed are counted as passed, the errored ones as errors, and none of them as done.

A group or a suite has no code of its own that runs; it holds tests, and each test runs in its own process, so an exit anywhere inside one test ends at most that test. The group or suite records it as ERROR and runs its other members as planned.

The one place an exit is meant to end the process is the script that runs the suite, after run() has returned. That is how a CI job or a script written by an AI agent turns the result into a process exit code:

if __name__ == "__main__":
	result = MyTest().run()
	sys.exit(0 if result.status is OrcraftStatus.PASS else 1)

Inside a component, reach for a checkpoint with is_error=True or assign error_description instead of calling exit(): the outcome is the same error, with a message that says why.

Checkpoints in setup and teardown

Inside a setup or teardown, a failing checkpoint is always treated as an ERROR, whether or not is_error was passed:

def _setup(self) -> None:
	self.checkpoint(
		os.path.isdir(self.config.app_folder),
		"App folder found.",
		f"App folder not found: {self.config.app_folder}",
	)

In a case body the same call would record a failure and carry on, because a case is expected to report every problem it finds. In a setup or teardown it aborts the scope instead - a precondition that does not hold is not a finding about the system under test, it is a reason not to test it yet. The failure is rendered as a clean error message rather than a traceback, since a lifecycle assertion failing is an expected outcome, not a crash.