Orcraft Test Step

A step is the smallest unit of work in Orcraft: one action, its inputs, its outputs. Steps exist for two reasons. They make a long case body readable by naming its parts, and they are the only component that can be dispatched to a remote endpoint - which is what turns a local test into a distributed one.

Like every component, a step is three classes:

from __future__ import annotations

import os

from orcraft import (
	OrcraftTestStep,
	OrcraftTestStepConfig,
	OrcraftTestStepResult,
	OrcraftConfigField,
	OrcraftResultField,
)


class MyStepConfig(OrcraftTestStepConfig):
	name = OrcraftConfigField.Text(default="My Step")
	output_folder = OrcraftConfigField.Text(default="", label="Output Folder")
	run_id = OrcraftConfigField.Text(default="", label="Run Id")


class MyStepResult(OrcraftTestStepResult):
	output_file = OrcraftResultField.Text(default="", label="Output File")


class MyStep(OrcraftTestStep):
	def __init__(self, config: MyStepConfig | None = None) -> None:
		super().__init__(config or MyStepConfig())

	def _run_step(self) -> OrcraftTestStepResult:
		self.result = MyStepResult()

		output_file = os.path.join(self.config.output_folder, f"{self.config.run_id}.dat")
		self.result.output_file = output_file

		was_removed = os.path.exists(output_file)
		if was_removed:
			os.remove(output_file)

		self.checkpoint(
			True,
			f"Stale output removed: {output_file}" if was_removed else "No stale output to remove.",
			"",
		)

		return self.result

Step body

_run_step is the only method a step must implement. It creates its result, does the work, and returns the result - and the returned object must be an OrcraftTestStepResult, which the framework checks rather than letting a mistyped return surface later as a missing attribute.

A step is not handed a case. That is the point: it receives everything through its config and reports everything through its result, with no access to the test around it. Keeping to that discipline is what lets the same step run locally during development and on a remote endpoint in a distributed run, unchanged.

Config fields

Every field a step config starts with, before you add your own. Only target_endpoint and expect_endpoint_disconnect are specific to steps; the rest is the base every component shares:

Overridable in code

Set in your config class, fixed for the run.

Reserved

Written by the framework while a run is in progress.

Result fields

A step creates its result inside _run_step and returns it. The fields you declare on it come from the OrcraftResultField namespace; the rest it inherits from OrcraftTestStepResult, and most of those are written by the framework rather than by you:

overridable by user (optional)  ·  written by the framework, may still be set directly  ·  reserved

Granularity

A useful step is one action that can succeed or fail on its own terms: delete a file, launch a process, read a register, transfer a build. If a step needs two checkpoints about unrelated things, it is probably two steps. If it exists only to pass a value from one step to the next, it is probably not a step at all.

Splitting at that grain pays off twice. The log gains a banner and a result table per action, so a failure points at the action that produced it, and each of those actions becomes independently reusable - the same step appears in several flows across different tests.

Remote execution

When a flow runs a step against a remote endpoint, the framework transfers the test package to that endpoint, executes _run_step there, and returns the result to the controller. The step code does not know the difference, which means two things in practice:

• Paths inside a step are paths on the machine the step is executing on. Anything a step needs must arrive as a package field or through the flow's transfer helper.
• Anything a step wants to hand to the next step must go through its result, since the two may not be running on the same machine. It arrives with its Python type intact - see Type fidelity.

Orcraft Endpoints covers dispatch, package upload and the endpoint field types in full.

Type fidelity

A flow passes values between steps as if every step ran in the same process, and the framework keeps that true across endpoints. Arguments are pickled to the endpoint and results are pickled back; lists, tuples and dicts are walked so nested structures keep their shape; enums and your own classes work because the test package is on the endpoint too. A tuple arrives as a tuple, not a list, and an enum member arrives as that member, not its string.

first_result = self.run_step(
	MyProduceStep,
	endpoint=self.config.endpoint_1,
)

if self.handle_errors(first_result, self.result):
	return

# channel_map is still a tuple, mode still an Enum member, calibration still a dict of
# floats - even though first_result came from endpoint_1 and MyConsumeStep runs on endpoint_2.
second_result = self.run_step(
	MyConsumeStep,
	endpoint=self.config.endpoint_2,
	channel_map=first_result.channel_map,
	mode=first_result.mode,
	calibration=first_result.calibration,
)

This is what lets a flow be written once and distributed later: the code that wires one step's output into the next step's input does not change when the steps move to different machines. The values usually travel as plain attributes on the result and plain attributes on the next config, since structured data is rarely something a person wants as a GUI column or input.

Non-transferable values

Some values have no meaning off the machine that made them - a thread, an open socket, a process handle. When a remote step puts one of those on its result, it does not come back as a value. It stays on the endpoint, and the flow receives a reference to it instead. That reference can be handed to a later step on the same endpoint, which gets the live object back; passing it to a local step, or to a different endpoint, raises. That keeps a long-lived resource usable across several steps without pretending it can be copied.

Synchronous execution

A step runs to completion before the flow continues. run_step returns only after _run_step has finished, on the controller or on the endpoint, and hands back the finished result, so the next line of the flow can read it. The steps of one flow never overlap, and _run_step is a plain method: an async def body is not awaited and is not supported.

Concurrency in Orcraft lives above the step. Cases run in parallel, and an endpoint fanout runs the same test on several machines at once, each with its own log and its own place in the tree. Inside a flow the order of steps is the order in which things happen, which is what makes the log and the tree readable when a step fails.

Overlapping work

When a flow does need two things in flight at the same time, split the long-running side into two steps: a start step that launches the work and returns, and a later wait step that collects its outcome. Between them the flow runs whatever must overlap, on the same endpoint or on another one. A process handle, thread or socket kept on the start step's result stays on the endpoint and reaches the flow as a reference, as described under Non-transferable values above; the wait step hands it back to the same endpoint and gets the live object. Each step stays one action with its own checkpoint, and nothing in the flow's contract changes.

Direct execution

Steps are normally run by a flow through run_step, which builds the step, applies the config values and dispatches it. A step can also be constructed and run directly, which is occasionally useful while developing one:

step = MyStep(
	MyStepConfig(output_folder="/tmp/run", run_id="0001")
)
result = step.run()

A step run this way always runs locally - remote dispatch is something the flow arranges.