Orcraft Test

A test is the smallest thing in Orcraft worth building on its own. It needs no group, no suite and no endpoint: a single Python file with a test class in it can be run from the command line, and the same file, unchanged, is what the controller loads later.

Every test is three classes - a config, a result and the test itself:

from __future__ import annotations

from orcraft import (
	OrcraftTest,
	OrcraftTestCase,
	OrcraftTestConfig,
	OrcraftTestResult,
	OrcraftConfigField,
	OrcraftResultField,
)


class MyTestConfig(OrcraftTestConfig):
	name = OrcraftConfigField.Text(default="My Test")
	description = OrcraftConfigField.Text(default="Verifies that the system under test responds correctly.")
	payload_size = OrcraftConfigField.Numeric(default=64, label="Payload Size")
	threshold = OrcraftConfigField.Numeric(default=100, label="Threshold")


class MyTestResult(OrcraftTestResult):
	measured_value = OrcraftResultField.Numeric(default=0, label="Measured Value")


class MyTest(OrcraftTest):
	def __init__(self, config: MyTestConfig | None = None) -> None:
		super().__init__(config or MyTestConfig())

	def _run_case(self, case: OrcraftTestCase) -> OrcraftTestResult:
		case.result = MyTestResult()

		measured = measure(payload_size=self.config.payload_size)
		case.result.measured_value = measured

		self.checkpoint(
			measured <= self.config.threshold,
			f"Measured {measured}, within the threshold of {self.config.threshold}.",
			f"Measured {measured}, above the threshold of {self.config.threshold}.",
		)

		return case.result


def main() -> None:
	MyTest().run()


if __name__ == "__main__":
	main()

Config class

The config declares everything about the test that can be configured from outside it. Each field is a class attribute built from the OrcraftConfigField namespace, and inside the test it reads as an ordinary attribute - self.config.payload_size - holding either the configured value or the declared default.

Subclassing OrcraftTestConfig means the config starts out with fields it did not declare. Overriding one, as name and description are overridden above, replaces its default without changing what it is for.

Overridable in code

Set in your config class, fixed for the run.

Overridable in code or GUI

Set in your config class, and changeable again from the run configuration panel before a run.

Fields sitting at their default - a single cycle, an empty matrix - are hidden from the run panel rather than shown as clutter, and the endpoint policy fields only appear for a test that actually uses endpoints.

upload_test_package takes one of three values:

update_endpoint_app takes one of three values:

Reserved

Written by the framework while a run is in progress.

Result class

The result declares what the test reports. Its fields come from the OrcraftResultField namespace, and like the config it starts out with fields it did not declare. Unlike the config, most of these are written by the framework, not by you:

overridable by user (optional)  ·  written by the framework, may still be set directly  ·  reserved

Declaring a result class is optional. A test that has nothing of its own to report can leave case.result alone - it is then a plain OrcraftTestResult, with the same inherited and reserved fields - and still produce a complete pass or fail. Declaring one is what makes measured values appear in the log tables, in the controller and in exported results.

Test body

_run_case is the only method a test must implement. The framework calls it once per case per cycle, hands it the case, and expects the case result back:

def _run_case(self, case: OrcraftTestCase) -> OrcraftTestResult:
	case.result = MyTestResult()
	...
	return case.result

Assigning case.result to your own result subclass is what puts the extra result fields in play - without it the case reports only the inherited ones. Everything else about the body is ordinary Python: call into the system under test however you like, record what you measured on case.result, and assert with checkpoint.

A case body does not have to be small. When it grows past what reads well in one method, split it into steps and flows - but that is a decision to make when the test needs it, not up front.

Cycles

Repetition is configuration, not code. Two independent counters on the test config control it, and both are editable from the controller before a run:

The difference is the order of execution. Test cycles repeat the sequence: every case once, then every case again. Case cycles repeat in place: the first case three times, then the second case three times. For a flaky interaction, case cycles hammer the same conditions; for wear over time, test cycles walk the whole matrix repeatedly.

A test with four cases, two test cycles and three case cycles executes the case body twenty-four times. Both counters are hidden from the run panel while they sit at one, so a test that does not repeat does not advertise the fields.

Cycles are a test-level idea. Suites, groups, flows and steps have no cycle counter of their own: a suite repeats because the tests inside it do, and a flow repeats because the case that calls it does. Orcraft Setup, Teardown and Exit gives each cycle scope its own setup and teardown when per-cycle state has to be rebuilt.

Setup and teardown

Work that must happen around the cases rather than inside them - launching an application, opening a connection, generating shared inputs - belongs in a lifecycle hook rather than at the top of _run_case:

def _setup(self) -> None:
	self.app = MyApplication(self.config.app_folder)
	self.app.start()

def _teardown(self) -> None:
	self.app.stop()

Every hook is optional. Orcraft Setup, Teardown and Exit covers all six of them and what happens when one fails.

Standalone execution

Giving the file a main() makes it directly executable, which is how a test is developed before it is added to a group:

def main() -> None:
	MyTest().run()


if __name__ == "__main__":
	main()

run() executes the full lifecycle - setup, every case of every cycle, teardown - and logs the config table, the per-case results and the test summary exactly as the controller would. There is no separate packaging step and no alternate entry point: the file that runs from a terminal is the file the controller loads.

Once the test does what you want, add it to a group to run it alongside its neighbors.