Orcraft Endpoints

An endpoint is a machine that executes steps on the controller's behalf. Registering them is a setup task - see Adding endpoints - and this page is about the other half: how a test says which endpoints it needs, what gets sent to them, and how a step ends up running there.

Endpoints are configuration, not code. A suite developed entirely against localhost becomes a distributed suite by changing values in the run panel, with no edit to the test body.

Endpoint fields

Every test config already declares test_endpoint, defaulting to localhost. A test that needs more than one, or a different kind of selection, declares its own:

class MyTestConfig(OrcraftTestConfig):
	test_endpoint = OrcraftConfigField.Endpoint(default="localhost", label="Test Endpoint")
	aux_endpoints = OrcraftConfigField.MultiSelectEndpoints(default=("localhost",), label="Aux Endpoints")

The difference between the last two is what happens when several endpoints are selected. MultiSelectEndpoints gives one run with several endpoints available to it - the test loops over them and decides what each one does. EndpointFanout gives several independent runs of the same test, one per endpoint, executing in parallel. Use fanout to run the same verification everywhere; use multi-select when the endpoints play different parts in one scenario.

Redeclaring test_endpoint is how a test changes its own dispatch model, because the base config already declares it as a plain Endpoint:

test_endpoint = OrcraftConfigField.EndpointFanout(
	default=("localhost",),
	label="Test Endpoint",
	shared=True,
)

Field options

By default an endpoint is held for the duration of a run, and another run wanting the same endpoint waits its turn - which is what you want when a test owns a device. shared=True lifts that, treating each run as an independent client, which is what you want when several fanout-expanded runs legitimately share one host.

Endpoint pools

A pool asks for any available endpoint instead of naming one. It is declared as the field's default, in place of a machine name, and the machines are picked when the run starts. The test declares its roles and how many endpoints each takes; which endpoints play them is decided by what the fleet has free at that moment:

class MyTestConfig(OrcraftTestConfig):
	test_endpoint = OrcraftConfigField.Endpoint(
		default=OrcraftConfigField.EndpointPool(
			OrcraftConfigField.Query.match("os", "Ubuntu 24.04"),
		),
		label="Test Endpoint",
	)
	device_endpoints = OrcraftConfigField.MultiSelectEndpoints(
		default=OrcraftConfigField.EndpointPool(
			OrcraftConfigField.Query.match("usb", "ftdi"),
			OrcraftConfigField.Query.distinct("os"),
			count=2,
		),
		label="Device Endpoints",
	)

Conditions

EndpointPool takes its conditions as positional arguments, the way Matrix takes its dimensions, and count as a keyword. The conditions combine as AND. A condition is a Query record of one of two kinds:

Attributes

The attributes are what the controller records about each machine. os is the operating system exactly as the endpoint reports it, such as Windows 11, Ubuntu 24.04 or macOS 26.2, and a condition names it in full: the one who declares the pool knows the fleet, and a test that cares about the operating system says which one. The other attributes are the labels a person gives an endpoint in the controller, such as gpu: discrete or usb: ftdi, and the machine facts chosen there, such as the architecture. Names and values compare regardless of case.

Count

What count means follows the field type:

Picking

The pick follows rules a test cannot change. localhost is never picked. A test's own endpoint, the one its test_endpoint holds, is never picked again for its other fields, and no test takes an endpoint twice. Among the qualifying endpoints the first free ones win, in the order the endpoints are listed; a qualifying endpoint that another run holds is waited for, as a named one would be, and the run shows as queued until the pick is made.

Refusals

A pool the fleet cannot fill is refused before anything runs, with a message that names the test and the field: the test's config names no endpoints file, no endpoint carries the attribute a condition names, fewer endpoints qualify than the count asks for, fewer distinct values exist than the count, or the test's pools together ask for more endpoints than the fleet can give at once.

In the run

The test body never sees a pool. By the time it runs, the field holds endpoint values in the field's own shape - one for Endpoint, a tuple for the other two - and the log's configuration table and the run history record the endpoints that were picked. In the controller the run panel shows the pool and may replace it for a run: other conditions, fewer candidates, another count, or endpoints picked by hand, as the tutorial's Endpoint Pools page shows.

Endpoints file

Outside the controller, the fleet a pool picks from is the controller's saved endpoints file, named by the test's own config in endpoints_file_path. A relative path is relative to the file of the config class that sets it:

class MyTestConfig(OrcraftTestConfig):
	test_endpoint = OrcraftConfigField.Endpoint(
		default=OrcraftConfigField.EndpointPool(
			OrcraftConfigField.Query.match("os", "Ubuntu 24.04"),
		),
		label="Test Endpoint",
	)
	endpoints_file_path = OrcraftConfigField.Text(default="../endpoints.json")

The test then picks the same way however it runs: on its own from the command line, in a group, a suite or a batch. Each test names its own file. In the controller the field is never shown: the controller picks from its own endpoints. A CI snapshot that holds a pool is exported with the session's file beside its batch, and each test that asks for a pool names that file, as the tutorial's Exporting Test Batch for CI page shows. A test without pools needs no file.

Step dispatch

Steps are the unit of remote execution. A flow chooses where each one runs:

produce_result = self.run_step(
	MyProduceStep,
	endpoint=self.config.endpoint_1,
	payload_size=self.config.payload_size,
)

consume_result = self.run_step(
	MyConsumeStep,
	endpoint=self.config.endpoint_2,
	input_file=transferred_path,
)

With no endpoint argument, a step runs on the endpoint the test is configured for. The framework establishes the connection, sends the step and its config, executes it there, and returns the result - so from the flow's point of view a remote step and a local one are the same call.

Because the step may run on another machine, everything it needs must arrive through its config, and everything it produces must leave through its result. A step cannot reach back into the test.

Package upload

Before a step runs remotely, the endpoint needs the code. The controller packages the suite and uploads it, governed by upload_test_package:

Suite requirements

A suite that imports packages beyond the Python standard library lists them in a requirements.txt at the root of its repository, in pip's own format. After the suite package is activated on an endpoint, the controller has the endpoint application's Python install that file with pip, once per uploaded package; a package the endpoint already holds costs one short check. The outcome appears in the log next to the upload lines and names the requirements. When the install fails, the test errors before any step runs and the log carries pip's message.

The FTDI example suite drives FT2232H modules through pyftdi, which needs a libusb library that Windows does not ship, so its file lists both:

pyftdi
libusb-package

The controller's own Python is never touched. A test that runs on localhost uses the controller's Python directly, so those packages must be installed there by hand. A suite without a requirements file skips the step entirely.

Data files are a separate matter from code. A file a test needs on an endpoint is declared as a package field, and the endpoints option binds it to the endpoint fields that actually need it:

input_file = OrcraftConfigField.PackageFile(
	default="",
	label="Input File",
	required=True,
	endpoints=[endpoint_1],
)

input_folders = OrcraftConfigField.PackageFolderList(
	default=(),
	label="Input Folders",
	required=True,
	endpoints=[test_endpoint, aux_endpoints],
)

Inside the step, the value resolves to the path on the machine the code is executing on. Without endpoints, a package field is uploaded to every endpoint the test uses - fine for a small binary, wasteful for a large data set that only one endpoint reads. The field types are listed in Configuration fields.

Endpoint application update

update_endpoint_app decides whether the endpoint application itself is brought up to date before the run:

File transfer

Package upload sends files from the controller. Moving files between endpoints during a run - produce on one machine, verify on another - is what a flow's transfer is for:

transfer_result = self.transfer(
	produce_result.output_file,
	src_endpoint=self.config.endpoint_1,
	dst_endpoint=self.config.endpoint_2,
)

if transfer_result:
	return

consume_result = self.run_step(
	MyConsumeStep,
	endpoint=self.config.endpoint_2,
	input_file=transfer_result.dst_paths[0],
)

paths takes a single file or folder path, or a list of file paths, and they travel as one zip archive rather than as a call per file. The returned object is truthy when the transfer failed, matching the handle_errors convention, and dst_paths holds one absolute path per element of paths, pointing at where each landed on the destination.

Endpoint state

Tests that deliberately disrupt an endpoint - a reboot, a service restart, a network drop - need to wait for it to come back. OrcraftEndpointState names the three states:

from orcraft import OrcraftEndpointState

...

self.run_step(
	MyRebootStep,
	endpoint=endpoint,
	expect_endpoint_disconnect=True,
)

if self.wait_for_endpoint_state(
	OrcraftEndpointState.OFFLINE,
	endpoint=endpoint,
	timeout_sec=self.config.disconnect_timeout_sec,
):
	return

if self.wait_for_endpoint_state(
	OrcraftEndpointState.READY,
	endpoint=endpoint,
	timeout_sec=self.config.connect_timeout_sec,
	poll_interval_sec=5,
):
	return

wait_for_endpoint_state returns truthy on failure, like the other flow helpers, so if ...: return reads as the error path throughout. restart_endpoint wraps the whole sequence - restart, wait for the disconnect, wait for readiness - for the common case.

The step that causes the disconnection must be marked expect_endpoint_disconnect=True. Without it, the connection dropping mid-step is exactly what an infrastructure failure looks like, and the framework reports it as an error.

Security

Communication with an endpoint is secured with mutual TLS and token-based authentication, negotiated from how the endpoint was registered with the controller. Tests do not handle credentials: the security context follows the endpoint field, and flows and steps - including those called from setup and teardown methods - inherit it automatically.

secure_enabled on an endpoint field overrides the negotiated setting for that field, which is mainly useful for tests that exist to verify the secured and unsecured paths themselves.