Technical details

This page describes what happens underneath a request. You do not have to understand any of it to use Orcraft from an AI agent; the AI agent does. It is here for when you want to see what the AI agent reads and writes on the way. The AI agent reads the endpoints file to know the machines, writes a configured copy of the test to fill in the values of one run, runs it through a batch and reads the log. None of it needs the controller GUI, and none of it touches the suite repository.

Endpoints file

The AI agent has to know which machines exist and how to address them. That knowledge lives in the controller. The Endpoints panel is where a team registers its machines and checks that each one answers. The panel's Save button writes them to a JSON file, by default in the Endpoints folder of the controller's working directory. Register the machines, save, and put the file in the run folder for the AI agent.

File format

{
  "endpoints": [
    {
      "type": "Host Name",
      "host_name": "lab-linux-1",
      "ip": "10.0.0.21",
      "resolved_hostname": "lab-linux-1",
      "port": 50051,
      "secure_enabled": true,
      "os_name": "Ubuntu 24.04",
      "version": "0.1.0",
      "accepted_duplicates": [],
      "labels": {"gpu": "discrete"},
      "facts": {"architecture": "x86_64", "build": "#45-Ubuntu SMP"},
      "attribute_facts": ["os"]
    },
    {
      "type": "IP",
      "host_name": "",
      "ip": "10.0.0.31",
      "resolved_hostname": "lab-win-1",
      "port": 50051,
      "secure_enabled": true,
      "os_name": "Windows 11",
      "version": "0.1.0",
      "accepted_duplicates": [],
      "labels": {},
      "facts": {"architecture": "AMD64", "build": "10.0.26100"},
      "attribute_facts": ["os"]
    }
  ]
}

Each entry is one registered machine. type is "Host Name" or "IP", depending on how the machine was added. A machine added by IP address has an empty host_name, and resolved_hostname carries the name the controller found for the address. os_name is the operating system the machine reported the last time it answered, such as Windows 11, Ubuntu 24.04 or macOS 26.2, and version is the Orcraft version installed on it. Both are empty for a machine that has not answered yet. labels holds what a person said about the machine in the controller, facts what the machine reported about itself, and attribute_facts names the facts the lab treats as attributes, the operating system unless the lab chose more; the Labels section of the tutorial describes them.

The value the AI agent puts in an endpoint field is the host_name, or the ip for a machine added by address, followed by :port when the port is not the default 50051. A request that names a machine by host name, by address or by operating system resolves through these fields. A request for any available endpoint of a kind does not name one at all; it becomes an endpoint pool, and the file is what the run picks from.

Loading into a controller

The same file loads back into any controller through the panel's Load button. One saved list therefore serves the team and its AI agents alike, and it is the only list of machines the AI agent works from.

Configuring a run

Every value a test exposes in the controller's run panel is a config field on its config class. A subclass that redeclares a field with a new default is a configured copy of the test, and the framework treats the copy exactly like the original. The controller's own CI export uses this mechanism for its mirror classes. The AI agent uses it in the same way.

Configured copy

For the request "Run the TCP Traffic group: Linux machines as fan-out, Windows machines as link partners", the AI agent writes this file into the run folder:

from orcraft import OrcraftConfigField
from traffic_lib.base_traffic_test import DirectionalTrafficConfig
from test_group_tcp.test_tcp_tx.test_tcp_tx import TestTcpTx


class DirectionalTrafficConfig(DirectionalTrafficConfig):
	sut_endpoint = OrcraftConfigField.EndpointFanout(
		default=("lab-linux-1", "lab-linux-2"),
		label="SUT Endpoint",
	)
	link_partner_endpoints = OrcraftConfigField.MultiSelectEndpoints(
		default=("lab-win-1", "lab-win-2"),
		label="Link Partner Endpoints",
	)
	nuttcp_tool_package = OrcraftConfigField.PackageFolder(
		default="/home/lab/orcraft/tools/nuttcp",
		label="Nuttcp Tool Package",
	)


class TestTcpTx(TestTcpTx):
	def __init__(self, config=None):
		super().__init__(config if config is not None else DirectionalTrafficConfig(name="TCP Transmit"))

What changed against the original:

• The SUT field was redeclared from a single Endpoint to an EndpointFanout. The test now runs once per machine listed, in parallel.
• The link partner field kept its MultiSelectEndpoints type and received the partner machines. Every run shares them.
• The tool package, a required field with no value in the code, was filled in. A standalone run has no run panel to ask for it.

Everything else is inherited: the test body, the other fields and the result fields. The subclasses keep the names of the classes they extend, as the CI export's mirrors do, so the file reads as the same test with its values filled in. The endpoint field types are described on the Endpoints page.

When a test has several endpoint fields, the request says which machine plays which part. In the request above, "as fan-out" and "as link partners" name the two endpoint fields of the traffic tests: the machine under test, with one run per machine, and the set of partners that every run shares. The AI agent finds the fields and their types in the test's config class.

Any available endpoint

For the request "Run TCP Transmit on any available Linux endpoint", the AI agent does not choose an endpoint from the file. It declares the field's default as an endpoint pool, the kind of endpoint and how many, and the run picks a free one that fits when it starts. A pool names the operating system as the file spells it, so the AI agent reads the os_name of the Linux entries and writes those:

class DirectionalTrafficConfig(DirectionalTrafficConfig):
	sut_endpoint = OrcraftConfigField.Endpoint(
		default=OrcraftConfigField.EndpointPool(
			OrcraftConfigField.Query.match("os", "Ubuntu 24.04"),
		),
		label="SUT Endpoint",
	)
	endpoints_file_path = OrcraftConfigField.Text(default="endpoints.json")

A pool asks about the attributes the entries carry: the operating system in full, the labels and the facts chosen as attributes. The configured copy names the endpoints file the pool picks from in endpoints_file_path, relative to the copy itself, so a copy in the run folder names the file beside it. The log's configuration table names the endpoint that was picked.

Groups, suites and batches

A group or a suite is configured the same way, through a subclass of its own config class. Whatever the request covers, one item or several, the configured copies are listed in a batch in the same run folder. The batch is what runs them.

Running a batch

A configured copy lives outside the suite repository, so it cannot import the test on its own. It is run through a batch whose items say where the code is: the run folder for the copy, and the suite repository for everything the copy imports and for the package that ships to the endpoints. A CI snapshot exported from the controller runs its mirrors in exactly this way.

The run needs the orcraft package installed in the Python the AI agent uses, as described in Getting started. The controller GUI is not started.

Batch file

The batch file sits in the run folder beside the configured copies:

from orcraft import (
	OrcraftConfigField,
	OrcraftTestBatch,
	OrcraftTestBatchConfig,
	OrcraftTestBatchItem,
)


class TcpTrafficBatchConfig(OrcraftTestBatchConfig):
	name = OrcraftConfigField.Text(default="TCP Traffic Fan-out")
	items = OrcraftConfigField.Items(default_factory=lambda: [
		OrcraftTestBatchItem(source_path="run_tcp_tx.py", import_roots=("../orcraft-test-suite-network",)),
		OrcraftTestBatchItem(source_path="run_tcp_rx.py", import_roots=("../orcraft-test-suite-network",)),
	])


class TcpTrafficBatch(OrcraftTestBatch):
	def __init__(self) -> None:
		super().__init__(config=TcpTrafficBatchConfig())


if __name__ == "__main__":
	TcpTrafficBatch().run()

Relative paths in an item are relative to the batch file.

• source_path names the configured copy to run. The batch finds the test class in it, and its name.
• import_roots puts the repository on the import path of the item's process, so the copy finds the test it subclasses. The batch cannot find the repository from the copy: it sits beside the run folder, not above it.
• The endpoints receive the repository of the test the copy extends, not the run folder: the batch finds it from the copy, which keeps the class name of the test it extends.

The batch is run from the run folder like any framework file. Its log is written to a Logs folder there.

Exit code

run() executes the whole batch and returns its result. The result's status is an OrcraftStatus. A file that has to give an exit code to a calling script maps it itself:

import sys

from orcraft import OrcraftStatus


def main() -> None:
	result = TcpTrafficBatch().run()
	sys.exit(0 if result.status is OrcraftStatus.PASS else 1)

Remote endpoints

A run that reaches a remote endpoint needs the same security material as the controller: the authorization token and the certificates. The framework reads the token from the ORCRAFT_CONTROLLER_AUTH_TOKEN environment variable, and the certificates from the machine's Orcraft profile folder, where the controller's Security tab generated them. On a machine where the controller has completed the security setup, the AI agent only needs the variable defined in the environment it runs in.

Reading the log

A run writes its log to a Logs folder in the working directory, one .ansi file per component. The children of a group, suite or batch are filed in a folder named after the parent's log file. The console shows the same text.

Log tables

The log is made of titled tables. The titles are what the AI agent searches for:

Statuses

A status is PASS, FAIL or ERROR. The Statuses page explains the difference between the last two, which is what an analysis turns on. The log of a group, suite or batch also carries one Started and one Finished line per test, with status, elapsed time and endpoints in fixed columns.

Analysis

The log is the complete record of the run, so the AI agent that ran the test can also analyze it: explain a failure from its step and case, compare two runs of the same test on two machines, or summarize a batch. The result of a run started this way lives in this log only. It does not appear in the controller's history or in its details dialog.

Export for Agents

Runs made from the controller can be handed to the AI agent the other way round. The details dialog of any scope, from one test run to the whole session, has an Export for Agents button beside the spreadsheet export. It writes one folder under the controller's Results_Export folder, or in the folder chosen in its dialog, with:

• summary.txt: the scope's configuration and results as plain text, written failure-first. A failed or errored row keeps its whole subtree of flows, steps and descriptions; a passed row stays one line with its parameters and result fields, and the subtrees of passed cases and passed setup and teardown stages are dropped. It ends with the list of runs in the scope and where each log is in the folder.
• logs/: the log of every run the dialog shows, laid out as under the controller's Logs folder, with the colour codes stripped so each file is plain text.

The AI agent reads the folder like the log of a run it started itself: the same tables, the same statuses. Nothing in the folder is a prompt or an instruction; what to ask about it is the request.