Orcraft Statuses

Every result in Orcraft carries a status that says how the run went. A case starts at PASS and is moved from there by what the test reports: a warning that leaves it passing, a failure, or an error that ends it without a verdict. This page uses the three Status tests of the tutorial suite's Statuses and Results group, one per outcome, and shows how each one looks in the execution tree, the log and the Details dialog.

The three tests share the same shape. Each runs the Maze application over a small matrix that mixes valid inputs with invalid ones: a misspelt slove command and out-of-bounds start coordinates. The application answers an invalid input with a message on its error output, and each test turns that message into a different status. The application is behaving correctly in every case; the tests are about the reporting.

Select Status: Pass with Warning and Status: Fail in turn and set their Maze Executable Folder. Status: Error takes a Maze Bin Folder instead, the folder that holds one subfolder per Maze version:

Its matrix names two versions, v3.0.0 and v4.0.0. Only the first exists on disk, which is what the test needs to produce an error. Click Update Plan for each of the three tests.

Pass with warning

Source directory:

The test runs the application and, when the run produced an error message, records it as a warning:

def _run_case(self, case: OrcraftTestCase):

	maze_app = MazeApp(
		self.config.maze_executable_folder,
		case.params["command"],
		width=case.params["width"],
		height=case.params["height"],
		start_row=case.params["start_row"],
		start_col=case.params["start_col"],
	)

	result = maze_app.run()

	if result.error:
		case.result.warning_description = result.error

	return case.result

Assigning warning_description attaches the message to the case without changing its status. Click Run on the test and confirm:

The run's row is drawn in yellow and its status reads Pass (+Warn). The counters tell the rest: 16 planned, 16 done, 16 passed, none failed. A warning is information attached to a passing case, and the tree counts it as a pass. What changes is where the run is surfaced: the Warn chip above the tree counts one test with warnings, and the progress bar in the Execution Status frame turns yellow.

Click Details for the run to see which cases carried the warning:

The two cases with a valid command and valid coordinates read Pass; the other fourteen read Pass (+Warn), and the message the application wrote is in each one's Warning Description column further right. The Items Filter at the bottom narrows the tree to either kind.

Fail

Source directory:

The only difference from the previous test is the field the message is assigned to:

	result = maze_app.run()

	if result.error:
		case.result.failure_description = result.error

	return case.result

Assigning failure_description moves the case to FAIL on its own; the test never sets the status by hand, so the reason always travels with it. Run the test:

The row turns red. The test reports 16 done, 2 passed and 14 failed, a pass rate of 12 %, and the Fail chip appears next to Warn. A failure is a finding about the application: the case ran to the end, compared what it saw with what it expected, and the two differed.

The Details dialog lists the finding per case:

The same two cases pass and the same fourteen are now failures, each with its message in the Failure Description column. Whether an unexpected message is a warning or a failure is the test's decision, and these two tests make opposite ones on purpose.

Error

Source directory:

This test sets no description at all. It builds the executable folder from the case's version and runs the application from there:

def _run_case(self, case: OrcraftTestCase):

	bin_folder = f"{self.config.maze_bin_folder.rstrip('/\\')}/{case.params['maze_version']}"

	maze_app = MazeApp(
		bin_folder,
		case.params["command"],
		width=case.params["width"],
		height=case.params["height"],
	)

	maze_app.run()

	return case.result

For v4.0.0 the folder does not exist, launching the executable raises an exception, and nothing in the test catches it. An uncaught exception is what produces ERROR: the case could not reach a verdict, so there is no finding about the application, only about the run. Run the test:

The row turns orange and an Error chip appears above the tree. The log shows the traceback of each failed launch, the Test Matrix results with both v4.0.0 cases marked ERROR, the message as the test's Error Description, and an Errors table that lists the errored cases with their descriptions. The counters read 4 planned, 2 done and 2 passed, with progress at 50 %: an errored case does not count as done, because it produced no result to count.

The error ends the case, not the test. Its cause may belong to that one case, or be sporadic and gone by the next one, so the framework goes on with the next case and lets it report its own outcome. The same holds for a call to exit() anywhere in a test; the framework reference's Calling exit() section goes through every place it can happen. Open the Details dialog:

The two v3.0.0 cases passed and both v4.0.0 cases errored, each with the message in its Error Description column. Every case was reached, but the errored ones are missing from Done, which is why the progress stays at 50 % rather than reporting the run as complete.

The distinction between fail and error is the one to keep in mind when writing a test. A failure is expected and means the application is wrong; an error is unexpected and means the test could not do its job. Orcraft counts them separately so that a suite with an infrastructure problem does not look like a suite with a quality problem.

Statuses in the tree

Select the Statuses and Results group to see the three runs together:

Each test run row is coloured by its status, and the group aggregates the counters of its tests: 64 planned, 34 done, 20 passed and 14 failed, a progress of 53 %. The Warn, Fail, Error and Not Run chips above the tree count tests by their latest status, and clicking one narrows the tree to those tests. The two remaining tests of the group, Checkpoints and Result Fields, have not run yet and count as not run.

These tests set a status with a single description. The next page, Checkpoints, checks several things in one case and records the outcome of each check in the log.