Orcraft framework overview

Component model

Orcraft models a test system as a hierarchy of components. Each level owns the level below it and contributes its own configuration and its own result.

Every level but the case is written the same way: a config class, a result class and the component class that pairs them. What differs is whether there is a method to run - the declared levels hold their children in a config field, and the framework does the rest.

The first five levels are the structural hierarchy: a batch gathers suites, groups and tests to run as one, a suite holds groups, a group holds tests, and a test expands into cases. Flows and steps are the procedural hierarchy: they are what a case body actually does. A case may call flows, a flow calls steps, and a step is the unit that can be dispatched to a remote endpoint.

Every level is optional above and below the one you need. A single test file with a main() runs on its own without a group or a suite, and a case that only needs a few lines of Python does not have to be split into flows and steps at all. The pages in this section follow that order of dependence: the building blocks first - configuration fields, statuses and result fields - then the test that uses them, and outward from there to steps, flows, endpoints, groups and suites.

Orcraft package

The Orcraft test framework is the plain-Python library your tests are written against. It ships with the controller and is imported from a single package:

from orcraft import (
	OrcraftTest,
	OrcraftTestConfig,
	OrcraftTestCase,
	OrcraftConfigField,
)

There is no domain-specific language, no runner conventions and no decorators to memorize. A test is a Python class that inherits from a framework base class, declares its configuration as class attributes, and implements one method. Everything else - discovery, configuration, distribution to endpoints, execution, logging and result collection - is what the controller does with that class.

Because a suite is ordinary Python, the same files run without modification: standalone from the command line, from the controller GUI, from the controller CLI in a CI pipeline, and from an AI agent that sets the configuration in a subclass. The Tutorial walks through building one step by step; this section is the reference for what the framework itself offers.

Config, component, result

Whatever the level, a component is always three classes - a config that declares its inputs, the component that holds its behavior, and a result that declares its outputs:

class MyTestConfig(OrcraftTestConfig):
	name = OrcraftConfigField.Text(default="My Test")
	payload_size = OrcraftConfigField.Numeric(default=64, label="Payload Size")


class MyTestResult(OrcraftTestResult):
	measured_value = OrcraftResultField.Numeric(default=0, label="Measured Value")


class MyTest(OrcraftTest):
	def __init__(self, config: MyTestConfig | None = None) -> None:
		super().__init__(config or MyTestConfig())

	def _run_case(self, case: OrcraftTestCase):
		case.result = MyTestResult()
		case.result.measured_value = measure(self.config.payload_size)
		return case.result

Those declarations are not just Python defaults. The controller reads them to build the run configuration panel in the GUI, to validate that required values are supplied before a run starts, and to render the config and result tables in the log. A field added to a config class appears in the GUI with no further wiring.

Public API

Everything the framework exposes is importable directly from orcraft. Each entry links to the page that covers it: