Orcraft Statuses

Every component result carries a status saying how it went and descriptions saying why. This page covers the statuses themselves and the mechanisms that set them - checkpoint, handle_errors and the description fields - before Orcraft Result Fields covers what else a result can carry.

Statuses

The distinction between FAIL and ERROR is the one that matters in practice. FAIL is an expected kind of failure: the component compared what it expected with what it observed, they differed, and it reported that - the system under test is wrong. ERROR is an unexpected failure: something raised - a missing file, a dead connection, a bug in the test itself - before a comparison could be made, so nothing has been established about the system under test either way. A suite where half the results are errors has an infrastructure problem, not a quality problem, and Orcraft counts the two separately so that difference is visible at a glance.

A result starts at PASS and is moved from there - by a failing checkpoint, by an assignment to one of the descriptions, or by an uncaught exception, which is what produces ERROR. A call to exit() counts as one: the component reports "exit() called by user" as its error description, without a traceback.

An error ends the component it happened in, not the run around it. A case that errors is counted as one executed, errored case, and the test goes on with the next case - the cause may be specific to that case, or sporadic and gone by the next one. The Orcraft Setup, Teardown and Exit page's Calling exit() section lists what an error ends at every place it can happen.

Setting a status

A test rarely assigns status directly. Most of the time it states a finding and lets the framework derive the status from it:

from orcraft import OrcraftStatus

...

def _run_case(self, case: OrcraftTestCase) -> OrcraftTestResult:
	case.result = MyTestResult()

	response = self.query_system(self.config.payload_size)

	# ERROR: the case cannot produce a finding at all, so say why and stop.
	if response is None:
		case.result.status = OrcraftStatus.ERROR
		case.result.error_description = "The system under test returned no response."
		return case.result

	case.result.measured_value = response.value

	# FAIL: a failing checkpoint records the finding and carries on.
	self.checkpoint(
		response.value <= self.config.threshold,
		f"Measured {response.value}, within the threshold.",
		f"Measured {response.value}, above the threshold of {self.config.threshold}.",
	)

	# FAIL: assigning a failure description has the same effect as a failed checkpoint.
	if response.code != 0:
		case.result.failure_description = f"The system reported code {response.code}."

	# Still PASS, but worth flagging.
	if response.retries:
		case.result.warning_description = f"Succeeded after {response.retries} retries."

	return case.result

Each status has its own route:

Note what the example does not do: it never sets OrcraftStatus.FAIL by hand. Going through a checkpoint or a description means the reason travels with the status, so the log and the controller show why the case failed and not merely that it did. Assigning failure_description on a passing result moves it to FAIL on its own, which is why the two are never set together.

Only ERROR is worth assigning explicitly, and only when the case has to stop early - it is the one status that says the finding is missing rather than negative.

Reading a status

Statuses compare as an enum, which is occasionally useful when a caller wants to branch on the outcome of a nested component rather than just propagate it:

flow_result = flow.run()

if flow_result.status is OrcraftStatus.ERROR:
	Logger.log("Skipping verification - the flow could not complete.", color=Logger.Color.YELLOW)
	return case.result

For the usual case - copy the outcome upward and stop if it failed - reach for handle_errors instead of comparing statuses by hand.

Checkpoints

checkpoint is the framework's assertion, available on every component. It takes a condition and the message to log for each outcome:

self.checkpoint(
	measured <= threshold,
	f"Measured {measured}, within the threshold of {threshold}.",
	f"Measured {measured}, above the threshold of {threshold}.",
	details="Measured over the full run, not a single sample.",
)

Both outcomes are logged, so the log records what was checked and not only what broke. This is what makes a passing Orcraft run readable as evidence rather than as silence.

A failing checkpoint sets the component to FAIL with the failure message as its description, and then continues. That is deliberate: one case can report every problem it found instead of only the first. Pass is_error=True when a failed check means the rest of the work would be meaningless - a missing file, an application that did not start - and the component is marked ERROR and the scope abandoned instead.

Inside a setup or teardown a failing checkpoint is always treated as an error, whether or not is_error was passed. A lifecycle assertion that does not hold means the work it was guarding should not proceed.

Warnings

A warning records something worth surfacing that does not make the result wrong - a retry that succeeded, output that was unusual but valid, a slower run than expected. It is raised by assigning warning_description on the result:

case.result.warning_description = f"Retried {retries} times before succeeding."

The status is unaffected - setting warning_description never moves a result off PASS - but the case is counted as a case with warnings, and that count aggregates up through the group and suite results. A run can therefore be entirely green and still tell you where to look.

Propagating statuses

Components do not raise their way up the hierarchy - they return results, and the caller decides. handle_errors copies status and descriptions from one result to another and returns True when the caller should stop:

flow_result = flow.run()

if self.handle_errors(flow_result, case.result):
	return case.result

This is the idiom to use after every flow call in a case and after every step call in a flow. It keeps the failure attached to the component that produced it while still surfacing it at every level above, so the controller can show which step of which flow of which case of which test failed - rather than a single stack trace at the top.

Signature

def handle_errors(self, source_result, target_result) -> bool

It copies four fields from source to target: status, failure_description, error_description and warning_description. The copy is unconditional - the last call wins. That is harmless for failures, because the idiom returns on the first one, but it means a warning carried by an earlier flow is overwritten when a later flow passes cleanly. Read warning_description off the earlier result before the next call if you need to keep it.

handle_errors is available on OrcraftTest and OrcraftTestFlow - the two components that call something beneath them. A step runs nothing below it, and groups and suites aggregate their children's results on their own, so none of them carries the method.