Endpoint Pools

Every test on the previous page was told its endpoints by name in the run panel. That is the right way to work on one machine's bench. In a lab that several people and a pipeline share, the endpoint that matters is whichever one is free and fits, and a test that asks for an endpoint by name waits for that one while others stand idle. An endpoint pool is the other way of filling an endpoint field: the test says what kind of endpoint it needs and how many, and the run picks them from the registered endpoints when it starts. The test declares its roles; the fleet decides who plays them.

A pool asks for an endpoint by what it is: its operating system, which every endpoint reports, and the labels a person gives it. This page first puts labels on the endpoints registered on Endpoints Management, then follows the Endpoint Pool test of the Endpoints group through its run.

Labels

A card tells what the controller knows about an endpoint: its address, its operating system, its version. What the machine behind it is beyond that, the GPU it carries, the USB device attached to it, the project it is reserved for, is for a person to say, and a label is how: a key and a value, such as gpu: discrete or usb: ftdi. Labels describe capabilities, what a test may ask an endpoint for.

The Add Endpoint dialog, and Edit on a card, has a Custom Labels section: a key, a value and Add make a chip, and the chip's cross removes it. To label many endpoints at once, the Endpoints panel has a Labels button in its header, beside Manage:

Click Labels:

The dialog is a grid: one row per remote endpoint, one column per label key, and a cell is the endpoint's value for that key, empty where it has none. Typing into a selected cell fills every other selected cell of the column, so one value lands on several endpoints in one go. Add Key adds a column for a new key, and a column left empty on every row is dropped when the dialog is accepted. A key or value typed in another case takes the spelling the fleet already uses, so a pool's conditions never list one value twice.

The OS column is different: it is what the endpoint reports, read-only, and no label may take its name. It is one of the machine facts, together with the architecture and the build, and the key list beside Add Key offers each fact as a read-only column. A fact shown as a column is an attribute a pool can ask about, like a label; the operating system is one from the start, the others when a lab needs them. The labels and the facts are saved with the endpoints by the panel's Save button, so a saved endpoints file carries everything a pool asks about.

The Endpoint Pool test

Source directory:

Endpoint Pool is the Multi-Select Endpoints test of the previous page with a pool as the default of both its fields. The generator asks for any endpoint running Ubuntu 24.04 or Windows 11, the operating systems exactly as the endpoints report them, and the solvers for any two endpoints with distinct operating systems:

test_endpoint = OrcraftConfigField.Endpoint(
	default=OrcraftConfigField.EndpointPool(
		OrcraftConfigField.Query.match("os", ("Ubuntu 24.04", "Windows 11")),
	),
	label="Test Endpoint",
)

aux_endpoints = OrcraftConfigField.MultiSelectEndpoints(
	default=OrcraftConfigField.EndpointPool(
		OrcraftConfigField.Query.distinct("os"),
		count=2,
	),
	label="Aux Endpoints",
)

A pool is a count and conditions. match keeps the endpoints whose attribute has one of the values named, and distinct makes the picked endpoints differ on the attribute; the conditions combine, and the framework reference lists what they can ask about. The operating system is the attribute every endpoint carries, named in full as the endpoint reports it; the labels given above are the others. The flows and steps of the test are those of the Multi-Select Endpoints test and know nothing of pools: by the time they run, the fields hold endpoints, as they always did.

The pool in the run panel

Select the test and set its Maze Executable Folder; the endpoint fields need nothing, because their defaults are the pools:

Each field shows its pool as text instead of an endpoint name: Any available: os = Ubuntu 24.04 | Windows 11 for the generator, Any 2 available: distinct os for the solvers. Click the edit button of Aux Endpoints:

The dialog is the one every endpoint field opens, with a pool header above the table. Any available is ticked, because the field's default is a pool; untick it and the dialog is back to choosing endpoints by name. Count is how many endpoints the run takes, offered on a multi-select field only. Below it is one row per attribute the registered endpoints carry, the operating system first and the labels after it, with a box per value: ticking values keeps only those endpoints, leaving a row alone imposes nothing, and the ≠ toggle beside the row's name is the distinct condition. The table lists the endpoints the conditions leave in, every one of them a candidate; untick an endpoint to keep it out of the pick, which is how a role is pinned to one or two known endpoints while the rest of the fleet is spared. The summary beside the header counts the candidates. Whatever the dialog settles on replaces the test's default for the plan, like any other field value.

The condition rows are there with Any available unticked too, as a filter: ticking values narrows the list to the endpoints that carry them, and you pick from what is left by hand, as on the previous page. The filter only narrows the view: an endpoint already selected that the filter leaves out stays listed, in amber and marked outside the filter, and the run still uses it until it is unticked. The distinct toggle belongs to a pool alone, and the filter is not kept with the plan; only a pool is.

A pool the fleet can never fill as it stands, say three distinct operating systems where two are registered, is a missing argument: the field is marked like an empty required field, a yellow triangle in the header carries the reason, and Run refuses the test the way it refuses one without its Maze Executable Folder.

Running with a pool

Leave the dialog as it is, update the plan and run:

The Test Run row names the endpoints the run was given, as many as its Endpoints column has room for: the first free Windows 11 or Ubuntu 24.04 endpoint for the generator, then two endpoints of distinct operating systems for the solvers, neither of them the generator's, because a test never gets the same endpoint twice and never localhost. The pick is made the moment the run starts; while the endpoints it needs are held by other runs, the row shows the pool's text and Queued, and takes the endpoints as they come free. Open the run's Details:

Under every case the generate steps ran on GMG-TEST-1, the Windows machine that came first among the candidates, and the solve steps on GMG-TEST-3 and GMG-TEST-5, one Ubuntu and one macOS endpoint, as the distinct condition asked. The configuration table keeps the pools as the default and plan values of the fields, so a run can always be traced back to what it asked for and what it was given.

A pool works the same way outside the controller. There the test picks from the saved endpoints file its config names in endpoints_file_path, whether it runs from the command line or in a batch an AI agent runs, and a CI snapshot that holds a pool carries that file beside its batch, as Exporting Test Batch for CI shows.

The next page, Packages Uploading, follows the package fields that carried the Maze application to these endpoints, and shows how a field is tied to the endpoints that need its files.