Orcraft Test Suite

A suite is the top of the test hierarchy - the thing the controller loads and a CI pipeline runs, on its own or as an item of a batch. It holds groups and nothing else, and like a group it is declared rather than coded.

from __future__ import annotations

from orcraft import (
	OrcraftTestSuite,
	OrcraftTestSuiteConfig,
	OrcraftTestSuiteResult,
	OrcraftConfigField,
)

from my_first_group.my_first_group import MyFirstGroup
from my_second_group.my_second_group import MySecondGroup


class MySuiteConfig(OrcraftTestSuiteConfig):
	name = OrcraftConfigField.Text(default="My Suite")
	description = OrcraftConfigField.Text(
		default="Covers the system under test end to end."
	)
	version = OrcraftConfigField.Text(default="0.0.1")
	groups = OrcraftConfigField.Groups(
		default_factory=lambda: [
			MyFirstGroup(),
			MySecondGroup(),
		]
	)


class MySuiteResult(OrcraftTestSuiteResult):
	pass


class MySuite(OrcraftTestSuite):
	def __init__(self) -> None:
		super().__init__(config=MySuiteConfig())
		self.result = MySuiteResult()


def main() -> None:
	MySuite().run()


if __name__ == "__main__":
	main()

Groups field

groups uses default_factory for the same reason tests does: it constructs objects, and a factory keeps two suite instances from sharing them.

The suite file is also where the suite's import graph comes together - each group imported from its own package. Keeping one group per folder, imported by its suite, is what lets the controller present the suite as a tree that matches the layout on disk.

Config fields

Overridable in code

Set in your config class, fixed for the run.

Field	Default	Meaning
`name`	`"Test Suite"`	Display name in the GUI, CLI and logs. Not editable from the GUI.
`description`	`""`	Free text describing what the suite does. Not editable from the GUI.
`version`	`""`	Suite version, recorded with the results.
`groups`	`[]`	The groups this suite runs.
`log_orientation`	`HORIZONTAL`	Orientation of the logged config table, or `OFF` to leave it out of the log.

Override carefully

Redeclare a field with the same OrcraftConfigField type and a new default - name = OrcraftConfigField.Text(default="My Suite"). A bare assignment such as name = "My Suite" is not an override: it shadows the field with a plain attribute, and the GUI, the log table and required validation no longer see it.

Reserved

Written by the framework while a run is in progress.

Field	Meaning
`start_time`	When the run started. Stamped by the framework at run time.
`log_tab`	Log indentation. Assigned by the framework at every level.

Refused by the framework

A class of your own that declares one of these names fails as soon as it is defined, with an error naming the field, and an assignment to one at run time raises the same error. The framework alone writes them.

Result fields

Like a group, a suite has no body: its result is written entirely by the framework as its groups complete, and an empty OrcraftTestSuiteResult subclass reports everything below. It aggregates three levels of counters at once:

overridable by user (optional) · written by the framework, may still be set directly · reserved

Inherited field	Overridden by user (optional)	Reserved	Meaning
`status`	–		`PASS`, `FAIL` or `ERROR`, derived from the children.
`failure_description`	–		Why the system under test was judged incorrect. Setting it moves a passing result to `FAIL`.
`error_description`	–		Why the component could not complete.
`warning_description`	–		Something worth flagging that does not change the status.
`finish_time`	–		When the component finished.
`elapsed_time`	–		How long it took.
`log_orientation`		–	Whether the result table is logged horizontally, vertically or not at all (`OFF`). Defaults to vertical.
`total_groups`, `passed_groups`, `failed_groups`, `error_groups`, `warning_groups`	–		Group counters, summed from the groups that ran.
`total_tests`, `passed_tests`, `failed_tests`, `error_tests`, `warning_tests`	–		Test counters, summed from every test in every group.
`total_cases`, `done_cases`, `passed_cases`, `failed_cases`, `error_cases`, `warning_cases`	–		Case counters, summed from every test in the suite: the planned total, the done cases and the passed, failed, errored and warned ones, each logged with its share of the total.

Override carefully

The only field that is yours here is log_orientation, and only its default: redeclare it with the same OrcraftResultField type - log_orientation = OrcraftResultField.Format(default=OrcraftResultField.LogTableOrientation.VERTICAL) - never as a bare value, which shadows the field so the log table no longer sees it.

Refused by the framework

A class of your own that declares one of the reserved names fails as soon as it is defined, with an error naming the field, and an assignment to one at run time raises the same error. Only log_orientation is meant to be redeclared, and only to change its default.

Carrying case counts all the way to the top is what makes a suite summary meaningful: two suites can both report ten passing tests while one executed ten cases and the other four hundred.

Execution

The same file serves every way of running:

Where	How
Standalone	Execute the suite file directly, via its `main()`.
Controller GUI	Add the suite file to the controller and configure the run from the panel.
CI	Run the batch of a CI snapshot exported from the GUI, on a machine with the controller installed and no GUI.
AI agent	Let an AI agent write a file that subclasses the suite with the fields it needs and run it from the command line, with no GUI involved.

The CI snapshot is what makes the third reproducible: it carries the configuration exactly as it was set in the GUI, so a pipeline reruns a configured suite rather than a differently configured one. See Setup for how the controller and its CI runner fit together.