Orcraft Test Batch

A batch sits above suites. It holds any mix of suites, groups and tests, runs them as one and summarizes them as one. It is what a CI snapshot exported from the controller puts on top of its mirrors, and what you write yourself to run several suites, possibly from several repositories, with a single command. Like a suite it is declared rather than coded.

from __future__ import annotations

from orcraft import (
	OrcraftConfigField,
	OrcraftTestBatch,
	OrcraftTestBatchConfig,
	OrcraftTestBatchItem,
	OrcraftTestBatchResult,
)

from my_suite import MySuite
from smoke.smoke_group import SmokeGroup


class NightlyBatchConfig(OrcraftTestBatchConfig):
	name = OrcraftConfigField.Text(default="Nightly")
	description = OrcraftConfigField.Text(
		default="Everything that must be green before a release."
	)
	items = OrcraftConfigField.Items(
		default_factory=lambda: [
			MySuite(),
			SmokeGroup(),
			OrcraftTestBatchItem(source_path="../orcraft-test-suite-network/test_suite_network.py"),
		]
	)


class NightlyBatchResult(OrcraftTestBatchResult):
	pass


class NightlyBatch(OrcraftTestBatch):
	def __init__(self) -> None:
		super().__init__(config=NightlyBatchConfig())
		self.result = NightlyBatchResult()


def main() -> None:
	NightlyBatch().run()


if __name__ == "__main__":
	main()

Items field

items uses default_factory for the same reason groups does.

Instance or reference

An item is one of two things:

What a reference adds

A reference does two things an instance cannot. Two repositories may both define a top-level package with the same name - a common package, say - and importing both into one interpreter would let one shadow the other; a reference keeps each item in its own interpreter all the way down. And a reference is only a path, so it can be pointed elsewhere before the run: ci_run.py re-roots each one when a pipeline checks the repositories out in another place. That is why a CI snapshot lists its items this way.

Reference fields

A reference needs only its file, one suite, group or test per file. The batch finds the rest the way the controller does when it is given a file: the class, its kind and its config's name, from a short process of its own that imports the file; the folders the file's imports need, read from the file itself; and the repository whose code endpoint uploads ship. That is the file's own repository, or, for a CI mirror or a configured copy that extends a class of the same name, the repository of that class.

Relative paths in a reference are relative to the batch file, so a folder holding a batch and its mirrors can be copied as a whole.

Config fields

Overridable in code

Set in your config class, fixed for the run. A batch has no run configuration panel: the controller shows a batch's items, never the batch itself.

Reserved

Written by the framework while a run is in progress.

Item paths

The CI export computes the paths of every item when it writes the snapshot. It knows the folder it writes into and the folder each loaded suite came from, and it stores the relative path from the first to the second in the item. Take a repository that keeps its suites and its exported snapshot together:

orcraft-validation/
  CI/
    Validation/
      batch.py
      ci_run.py
      test_suite_controller/            <- mirrors written by the export
  Suites/
    test-suite-controller/              <- the suite as the team wrote it
      test_suite_orcraft_controller_validation.py

The snapshot was written into CI/Validation, so the item carries the way from there to Suites/test-suite-controller:

OrcraftTestBatchItem(
	source_path="test_suite_controller/test_suite_orcraft_controller_validation.py",
	import_roots=(".", "../../Suites/test-suite-controller"),
)

source_path points at the mirror beside the batch. import_roots puts the snapshot folder and the suite folder on the import path, and the suite folder, the repository of the class the mirror extends, is what uploads ship to the endpoints. Nothing is edited by hand: as long as the snapshot and the suites stay where they were at export time, python batch.py runs. When a pipeline checks the suites out somewhere else, ci_run.py re-roots the items by repository name, from --suites-root or from an ORCRAFT_SUITE_ROOT_<REPO> variable, as Batch execution below describes.

Batch execution

The same file serves every way of running:

A batch always runs its items concurrently, each in its own subprocess. Suite and group items start at once and start their own children; the tests of the whole batch, test items included, take their turn in one queue kept for the run, in the order the items list them, and a test starts when every endpoint it needs is free of the batch's other tests, exactly as it would under the controller. When an item finishes, its whole log is appended to the batch log as one block, so the batch log reads as nested blocks: the batch header, then each item's complete log as it finishes, then the batch summary.

A batch cannot contain another batch.

Result fields

A batch has no body: its result is written entirely by the framework as its items complete, and an empty OrcraftTestBatchResult subclass reports everything below. Items may be of different kinds, so the batch counts what every kind has, tests and cases, plus the items themselves.

overridable by user (optional)  ·  written by the framework, may still be set directly  ·  reserved

After its result table the batch logs an Items table, one row per item with that item's status, descriptions, times and counters, and then its warnings, failures and errors tables with an Item column in front, so a failing case can be traced to the item, group and test it belongs to.