Orcraft Test Group

A group collects related tests. Unlike a test, a group has no body: it is declared, not coded. Its config names the tests it holds, and the framework does the rest.

from __future__ import annotations

from orcraft import (
	OrcraftTestGroup,
	OrcraftTestGroupConfig,
	OrcraftTestGroupResult,
	OrcraftConfigField,
)

from my_group.my_first_test.my_first_test import MyFirstTest
from my_group.my_second_test.my_second_test import MySecondTest


class MyGroupConfig(OrcraftTestGroupConfig):
	name = OrcraftConfigField.Text(default="My Group")
	description = OrcraftConfigField.Text(
		default="Groups the tests that cover one area of the system under test."
	)
	tests = OrcraftConfigField.Tests(
		default_factory=lambda: [
			MyFirstTest(),
			MySecondTest(),
		]
	)


class MyGroupResult(OrcraftTestGroupResult):
	pass


class MyGroup(OrcraftTestGroup):
	def __init__(self) -> None:
		super().__init__(config=MyGroupConfig())
		self.result = MyGroupResult()


def main() -> None:
	MyGroup().run()


if __name__ == "__main__":
	main()

Tests field

tests uses default_factory rather than default because it constructs objects. A factory is called once per config instance, so two group instances never end up sharing the same test objects - a mistake that a plain default list would make easy and hard to spot.

The result class is usually empty. Subclassing OrcraftTestGroupResult without adding anything is still worth doing: it gives the group a result type of its own to extend later, and keeps the three-class shape consistent across the suite.

Purpose

A group is the unit the controller lets you act on as a whole:

• Configure once, apply to many. Setting a parameter at group level applies it to every test in the group, so a shared executable folder or endpoint is entered once rather than per test.
• Run a slice. A group can be run without the rest of the suite, which is how a focused rerun stays focused.
• Aggregate reporting. Group results carry both test counters and case counters, so a group answers how many of its tests came back clean and how many cases that took.

The tutorial works through the first two in Running a group.

Config fields

Overridable in code

Set in your config class, fixed for the run.

Reserved

Written by the framework while a run is in progress.

Result fields

A group has no body, so its result is written entirely by the framework as its tests complete. An empty OrcraftTestGroupResult subclass therefore reports everything a group needs; declaring fields of your own is for the rare group that computes something across its tests.

overridable by user (optional)  ·  written by the framework, may still be set directly  ·  reserved

Concurrent execution

A group runs its tests as concurrent subprocesses, which is what makes a suite spread across several endpoints finish in a useful amount of time. The order in which tests start belongs to the run as a whole, not to the group: the run keeps one queue over every test of every group, in declaration order, the way the controller keeps one queue for a session. A test starts as soon as every endpoint it needs is free of the run's other tests; until then it waits without holding a process, and the group log notes which endpoints it is waiting for. A waiting test does not hold back the tests after it. A test that declares no remote endpoint starts at once.

Endpoint contention is therefore handled for you: an endpoint is held for the duration of a run, and another run wanting it waits, unless the endpoint field is declared shared.

Standalone execution

Like a test, a group runs on its own:

def main() -> None:
	MyGroup().run()


if __name__ == "__main__":
	main()

This runs every test in the group as concurrent subprocesses, and logs a group summary after the individual test results. Once a group is doing what you want, add it to a suite.