Checkpoints

A checkpoint is the framework's assertion. It takes a condition and a message for each outcome, logs the outcome either way, and moves the case to FAIL when the condition does not hold. The Status tests set a status from a single observation; the Checkpoints test of the same Statuses and Results group checks several distinct things about every run of the Maze application and records each check on its own.

Select the Checkpoints test and set its Maze Executable Folder. Its matrix is the one the Status tests use: a valid and a misspelt command, two heights, and start coordinates that are either valid or out of bounds, 16 cases in all. Click Update Plan.

Defining checkpoints

Source directory:

The test body hands the case to the suite's Maze helper, which runs the application and then checks its output:

def _run_case(self, case: OrcraftTestCase):

	maze_app = MazeApp(
		self.config.maze_executable_folder,
		case.params["command"],
		width=case.params["width"],
		height=case.params["height"],
		start_row=case.params["start_row"],
		start_col=case.params["start_col"],
	)

	maze_app.test(case)

	return case.result

The helper's test method runs the application once and passes its output through three checks:

def test(self, case: OrcraftTestCase) -> None:
	result = self._execute()
	self._test_version_logging(result.stdout, case)
	self._test_path_found_message(result, case)
	self._test_error_messages(result.stderr, case)

Each check ends in a call to case.checkpoint. The first confirms that the application printed its version banner and that the banner matches the executable's version. The second applies to a solve that produced no error: it confirms that a path was reported and that the number of arrows in the drawn path agrees with the step count the application claims. The third compares the application's error output with the message the helper expects for the case's inputs:

def _test_error_messages(self, stderr: str, case: OrcraftTestCase) -> None:
	expected = self._get_expected_error()
	failure = None if expected in stderr else "Error message is not as expected."
	pass_msg = "Error message matches expected." if stderr else "No error messages as expected."
	fail_msg = failure or ""
	details = f"Expected error message:\n\n{expected}" if failure is not None else None
	case.checkpoint(failure is None, pass_msg, fail_msg, details)

The arguments are the condition, the message to log when it holds, the message to log when it does not, and an optional details text that is logged with a failure. A valid input is expected to produce no error output at all, so the passing message differs depending on whether the application wrote anything.

Checkpoints in the log

Click Run and confirm. When the run finishes, type No error messages as expected into the log's search field and click Next to jump to the first case:

Below the solved maze and its Path found line, four Checkpoint Pass entries record what was checked: the version banner matched, the path-found message was present, the arrow count matched the step count, and no error message appeared. A passing case is therefore readable as evidence of what was verified, not only as a green status.

Now search for (valid: 0 to 4):

This case asked to solve a maze from column -2. The application refused with Start (0,-2) is out of bounds for a 15×5 maze, so the version banner check still passes, and there is no path-found check because the run produced an error. The error message check fails: the helper expected Start col -2 is out of bounds (valid: 0 to 4), and the application's wording differs. The log shows the red Checkpoint Fail entry with the failure message, followed by a Checkpoint Failure Details block carrying the details text.

A failing checkpoint marks the case FAIL, records its message as the case's failure description, and then continues with the remaining checks. One case can therefore report every problem it found rather than only the first one. When a failed check means the rest of the work would be meaningless, is_error=True turns it into an ERROR that ends the case instead; that option, and how checkpoints behave inside setup and teardown hooks, is covered in the framework reference.

The outcome

Clear the search field. The end of the log holds the test's result and its failures table:

The test reports 16 done, 10 passed and 6 failed. The six failures are exactly the solve cases with a negative start coordinate: the application rejects the coordinate, but with a different message than the helper expects. The slove cases pass, because the application's complaint about the unknown command is the one expected, and the two fully valid cases pass every check. The failure description of the test as a whole is the message of the first failing checkpoint.

Click Details to see the same outcome per case:

Cases 2, 3, 4, 6, 7 and 8 read Fail with the checkpoint's message in their Failure Description column; the rest read Pass. A checkpoint thus does two jobs at once: it decides the status, and it leaves a line in the log that says what was checked and what was seen.

Checkpoints report whether a case is right. The next page, Result Fields, shows how a case reports the values it measured alongside its status.