Exporting Test Batch for CI

Everything configured in the controller so far lives in its session: which items are loaded and planned, the values in their run panels, the selected cases, the endpoints. A CI pipeline has no controller window, so it needs the same configuration in a form it can run on its own. Export for CI writes that form: a snapshot folder holding a batch, a framework file that lists the exported items with their configured values baked in, together with a launcher that runs the batch without any GUI and exits with a code a pipeline can gate on.

The export mirrors what is configured rather than copying the suites. For every exported suite, group and test it writes a small mirror class that subclasses the original and overrides only what was configured: which groups a suite runs, which tests a group runs, and the field values of each test. The suites' own code stays in their repositories and is imported from there, so an export never edits or duplicates a test. This is also the way an AI agent runs tests: it writes the same kind of file by hand.

A snapshot is not tied to one suite. The session can hold any mix of items, a whole suite, a group file, a single test file, suites from different repositories, and the export takes them all as they are planned. That is what makes it possible to keep a few snapshots for different purposes, a quick gate, a nightly regression, a full validation, each composed from whatever tests belong to it.

Composing the session

This page composes a regression set from three kinds of item and two repositories. Start with the tutorial suite loaded, then use Add Item three more times:

the Demo suite from its own repository. This suite is an internal one, used here only to show a second repository in the session; it is not provided with Orcraft. Any suite of your own in its own repository plays the same part;
the Structure and Lifecycle group file from the tutorial suite, as a group item of its own;
the Hello World test file from the tutorial suite, as a test item of its own.

Each is a top-level row in the tree. The set to export is then picked by selecting rows: hold Ctrl and click the Intro and Statuses and Results groups of the tutorial suite, the Basic group of the Demo suite, the Structure and Lifecycle group item and the Hello World test item. A suite need not go in whole; picking two of its groups takes those two:

Selecting more than one row opens the Multi-Test Selection panel in the right pane, the configuration layout for a set of tests (the Execution Tab page covers the tree's selection and action frames). Its upper part is a grid: one row per test in the selection, grouped under their suites and groups, and one column per configuration field any of them declares, so a field a test does not have shows as a dash. The lower part is the ordinary run panel of the test whose row is chosen in the grid.

The Maze Executable Folder column is where the status and lifecycle tests need their value. Browse the folder on one row, then right-click the cell and choose Apply this value to all tests: every test that has the field takes it. The Status: Error test has a Maze Bin Folder instead, set on its own row:

The orange dot on the edited cells marks values that differ from the plan. Click Update Plan to commit them for every test in the selection at once:

The values turn blue as planned values, and the panel's Export for CI button exports exactly this selection. The button is on every run panel and exports the scope the panel shows: one test from a test's panel, a group from a group's panel, the selected rows from a selection's panel, and everything planned from the session row.

Naming the snapshot

Click Export for CI. The export validates the configuration the way Run does, so a missing required field stops it with the same message, and then a dialog asks where to write the snapshot. It opens in the controller's ci folder with a name of the form ci__snapshot__2026-09-12__19-42-31.996, a timestamp that keeps every export apart.

A snapshot that has a purpose deserves a name that says so. Replace the proposed name with the purpose of the set, here regression; other snapshots might be called gate, basic, full-validation or nightly. The batch takes the folder's name, and it is the name the pipeline, the tree and the logs will show from now on. When a test is added to the set or a value is changed, compose the session again and export to the same name: the snapshot is rewritten in place, and the pipeline that runs it needs no change.

When the export is done, a message counts the exported items. The folder holds the batch, the launcher and one mirror package per item:

C:\orcraft\ci\regression\
  batch.py
  ci_run.py
  orcraft_test_suite_tutorial\
  orcraft_test_suite_demo\
  orcraft_test_suite_tutorial_2\
  orcraft_test_suite_tutorial_3\

Two of the packages come from the tutorial suite's repository as well, one for the group item and one for the test item, so they are numbered to keep them apart.

The batch

batch.py is the batch itself, a plain framework file with one item per exported top-level row:

class BatchConfig(OrcraftTestBatchConfig):
	name = OrcraftConfigField.Text(default='regression', label="Name")
	items = OrcraftConfigField.Items(default_factory=lambda: [
		OrcraftTestBatchItem(
			source_path='orcraft_test_suite_tutorial/test_suite_tutorial.py',
			import_roots=('.', '../../orcraft-test-suite-tutorial'),
		),
		OrcraftTestBatchItem(
			source_path='orcraft_test_suite_demo/test_suite_demo.py',
			import_roots=('.', '../../orcraft-test-suite-demo'),
		),
		OrcraftTestBatchItem(
			source_path='orcraft_test_suite_tutorial_2/test_group_structure_and_lifecycle/test_group_structure_and_lifecycle.py',
			import_roots=('.', '../../orcraft-test-suite-tutorial'),
		),
		OrcraftTestBatchItem(
			source_path='orcraft_test_suite_tutorial_3/test_group_intro/test_hello_world/test_hello_world.py',
			import_roots=('.', '../../orcraft-test-suite-tutorial'),
		),
	])

Each item names the mirror it starts from in source_path, and import_roots puts the snapshot and the suite's own folder on the import path. The batch finds the rest from the mirror: the class, its name, and the suite folder that is uploaded to the endpoints, the repository of the class the mirror extends. The two suites point at two different repositories, and the group and the test point back at the tutorial's. The paths are relative to where the snapshot was written, which is why the suites have to stay where they were, or be pointed at again, when the snapshot runs elsewhere.

A suite's mirror is a few lines. It imports the original config class, overrides its groups field with the groups that stayed planned, and marks itself as a mirror:

from orcraft import OrcraftConfigField
from orcraft import OrcraftTestSuite
from test_suite_tutorial import TestSuiteTutorialConfig
from orcraft_test_suite_tutorial.test_group_intro.test_group_intro import TestGroupIntro
from orcraft_test_suite_tutorial.test_group_statuses_and_results.test_group_statuses_and_results import TestGroupStatusesAndResults

class TestSuiteTutorialConfig(TestSuiteTutorialConfig):
	groups = OrcraftConfigField.Groups(default_factory=lambda: [TestGroupIntro(), TestGroupStatusesAndResults()])

class TestSuiteTutorial(OrcraftTestSuite):
	orcraft_ci_mirror = True

	def __init__(self):
		super().__init__(config=TestSuiteTutorialConfig())

The group mirrors below it do the same for their tests, and each test mirror redeclares the fields that were configured, such as the Maze Executable Folder, with the values from the panel. The framework reference describes the item paths and the mirrors in full.

Running the snapshot

ci_run.py is the launcher. It runs the batch with no GUI, resolves where each item's repository is, and turns the outcome into an exit code. Its own header documents the command line:

Usage:
    python ci_run.py [--suites-root DIR]

Arguments:
    --suites-root   Directory that holds each suite's source repo, one folder per
                    repo (matched by folder name). Highest-priority way to point
                    the snapshot at the code it needs.

Endpoint pools: when a test asks for any available machine, the export wrote
the session's endpoints file as endpoints.json beside this launcher, and that
test's mirror names it in its endpoints_file_path; replace the file to point
the snapshot at another fleet without re-exporting.

Suite-root resolution (first match wins), per item:
    1. --suites-root DIR         -> DIR/<repo_name>
    2. ORCRAFT_SUITE_ROOT_<REPO> -> that path (REPO is the upper-cased repo name)
    3. the relative path baked in at export time (works when the snapshot sits
       next to the original repos).

Exit codes: 0 = all passed, 10 = a failure, 20 = an error.

With two repositories in the batch, a pipeline that checks them out side by side under one folder passes that folder as --suites-root, and each item finds its repository by name. A snapshot in which a test asks for any available machine also carries the endpoints to pick from: the export writes the session's endpoints, as the panel's Save button would, to endpoints.json beside the batch, and the mirror of each such test names that file in its endpoints_file_path. The pipeline's fleet can thus differ from the one the snapshot was composed with by replacing one file. Open a terminal in the snapshot folder and run the launcher with the Python that has the orcraft package installed:

python ci_run.py

The batch starts its four items in subprocesses, and each item's tests start together, exactly as they would under the controller. The console shows one line per test as it starts and finishes, with the item it belongs to:

Running 4 items in parallel subprocesses
	Test Suite: Orcraft Tutorial
	Test Suite: Demo
	Test Group: Structure and Lifecycle
	Test: Hello World

[2026-09-12  20:35:37.696]  Started   Test Suite: Orcraft Tutorial         Test Group: Statuses and Results  Test: Status: Pass with Warning                                                           localhost
[2026-09-12  20:35:37.697]  Started   Test Suite: Orcraft Tutorial         Test Group: Statuses and Results  Test: Status: Fail                                                                        localhost
[2026-09-12  20:35:37.697]  Started   Test Suite: Orcraft Tutorial         Test Group: Statuses and Results  Test: Status: Error                                                                       localhost
[2026-09-12  20:35:37.697]  Started   Test Suite: Orcraft Tutorial         Test Group: Statuses and Results  Test: Checkpoints                                                                         localhost
[2026-09-12  20:35:37.697]  Started   Test Suite: Orcraft Tutorial         Test Group: Statuses and Results  Test: Result Fields                                                                       localhost
[2026-09-12  20:35:37.759]  Started   Test Group: Structure and Lifecycle  Test: Steps and Flows                                                                      localhost
[2026-09-12  20:35:37.759]  Started   Test Group: Structure and Lifecycle  Test: Setup and Teardown                                                                   localhost
[2026-09-12  20:35:37.759]  Started   Test Group: Structure and Lifecycle  Test: Cycle Setup and Teardown                                                             localhost
[2026-09-12  20:35:37.759]  Started   Test Group: Structure and Lifecycle  Test: Case Setup and Teardown                                                              localhost
[2026-09-12  20:35:37.765]  Started   Test: Hello World                                                                             localhost
[2026-09-12  20:35:37.783]  Started   Test Suite: Demo                     Test Group: Basic                 Test: Hello World                                                                         localhost
[2026-09-12  20:35:37.783]  Started   Test Suite: Demo                     Test Group: Basic                 Test: Basic Parameters                                                                    localhost
[2026-09-12  20:35:37.783]  Started   Test Suite: Demo                     Test Group: Basic                 Test: Run Dialog Layout                                                                   localhost
[2026-09-12  20:35:37.783]  Started   Test Suite: Demo                     Test Group: Basic                 Test: Status: Fail                                                                        localhost
[2026-09-12  20:35:37.783]  Started   Test Suite: Demo                     Test Group: Basic                 Test: Status: Error                                                                       localhost
[2026-09-12  20:35:37.783]  Started   Test Suite: Demo                     Test Group: Basic                 Test: Status: Pass with Warning                                                           localhost
[2026-09-12  20:35:37.783]  Started   Test Suite: Demo                     Test Group: Basic                 Test: Extended Results                                                                    localhost
[2026-09-12  20:35:37.783]  Started   Test Suite: Demo                     Test Group: Basic                 Test: Log Tables Layouts                                                                  localhost
[2026-09-12  20:35:37.787]  Started   Test Suite: Orcraft Tutorial         Test Group: Intro                 Test: Hello World                                                                         localhost
[2026-09-12  20:35:38.181]  Finished  Test: Hello World                      PASS     0.003 s                          1 of 19      localhost

When the last test finishes, the items' logs are appended and the batch prints its result:

                   Test Batch: regression | Result                    

Parameter           | Value
----------------------------------------------------------------------
Name                | Test Batch: regression
Status              | ERROR
Error Description   | Intentional Exception for demonstration purposes
Failure Description | Intentional Failure for demonstration purposes
Warning Description | Intentional Warning for demonstration purposes
Finish Time         | 2026-09-12  20:35:51.677
Elapsed Time        | 14.441 s
Total Items         | 4
Passed Items        | 2
Failed Items        | 0
Error Items         | 2
Items With Warnings | 2
Total Tests         | 19
Passed Tests        | 14
Failed Tests        | 3
Error Tests         | 2
Tests With Warnings | 3
Total Cases         | 95

The batch reports what its items are built to produce: two items passed, two ended in an error, since both suites carry a test that deliberately errors, and the failing and warning tests are counted alongside. An error anywhere makes the batch an error, so the launcher exits with code 20; a run with failures and no errors exits with 10, and a clean run with 0. A pipeline step that runs python ci_run.py therefore fails exactly when the tests say it should, and the batch log written under the snapshot's Logs folder holds every test's log for the investigation.

Loading the batch in the controller

A snapshot is not only for pipelines. Add Item opens batch.py like a suite file, and the batch appears in the tree with its items beneath it:

The Test Batch row carries the snapshot's name, and beneath it the suite is shown in full, so the batch can be read in the context of the suite it came from. A cyan CI badge marks the tests that were configured and exported: Hello World and the five status tests here, and none of the tests of the groups that were unplanned before the export. The badge is the sign that a test's values come from the snapshot rather than from a panel.

Click the CI chip beside the level chips above the tree to see only those tests:

The tree now shows exactly the set that ci_run.py runs: the two suites narrowed to their planned groups, the group item and the test item. The batch runs, plans and reports like any other row, so a snapshot that failed in a pipeline can be opened here, run again, and inspected in the Details dialog with the same values the pipeline used.

The run the launcher made left more than logs behind: a run history file that the controller can open, so the pipeline's run shows in the tree with its rows and results as if it had run here. The next page, Run History, brings it in.