Orcraft 1.1.3 public beta released 08-Nov-2026. The Free Plan and the Team trial are open; card payments and invoices open shortly.

Orcraft

Orcraft 1.1.3 public beta released 08-Nov-2026. The Free Plan and the Team trial are open; card payments and invoices open shortly.

Orcraft

The platform for test development and orchestration
that lets you and your AI Agents run the entire testing operation

Watch the demo

Suite by an AI Agent

The two other examples were developed with an AI agent too, as every Orcraft suite can be. This page shows exactly how that goes: an AI agent writes a suite in one session, from the same technical note the Network Suite was built from and from the documentation on this site, following the requests the Example prompts page describes: a request that creates the suite and runs it, a fix when the run shows a problem, and a development request that changes what the suite exposes. Nothing on this page is staged. The AI agent kept a journal of every step as it worked, and this page follows that journal. The controller screenshots are of the suite it wrote.

Inputs

The AI agent started in an empty working folder with exactly the inputs the Inputs page lists, and nothing else:

C:\orcraft\agent-example\
	instructions.md                        the instructions file
	docs\Orcraft_Network_Test_Setup.pdf    the technical note of the setup under test
	orcraft-runs\endpoints.json            the machines, saved from the controller's Endpoints panel

The instructions file is the template from the Inputs page, which points the AI agent at the documentation, with its Paths section filled in for this machine:

## Paths

- Suite repositories: `C:\orcraft\agent-example\` (create a new suite repository here as a folder named after the suite)
- Run folder: `C:\orcraft\agent-example\orcraft-runs`
- Endpoints file: `C:\orcraft\agent-example\orcraft-runs\endpoints.json`
- Nuttcp Package Folder field: `C:\orcraft\apps\NUTTCP\bin` (holds `windows\nuttcp.exe`, `linux\nuttcp` and `macos\nuttcp`)
- Python with the orcraft package installed: `C:\orcraft\venv\Scripts\python.exe`
- Documents of the system under test: `C:\orcraft\agent-example\docs\`

The endpoints file is the lab's four machines as the controller saved them, GMG-TEST-1, GMG-TEST-2, GMG-TEST-3 and GMG-TEST-5, in the format the Technical details page shows. The note is the same four-page technical note that ships with the Network Suite repository, and the instructions file ships next to it there as docs/instructions.md. The nuttcp package folder is the one the Network Suite page describes.

The request

A getting started request names the source and what to build, and here also where to run it:

Create a suite for the network test setup described in docs\Orcraft_Network_Test_Setup.pdf: a TCP
Traffic group and a UDP Traffic group, each with a transmit, a receive and a bidirectional
throughput test, laid out like the tutorial suite. Then run it with GMG-TEST-1 as the SUT and GMG-
TEST-2 and GMG-TEST-3 as link partners, 10 seconds per case, and report the results from the log.

The AI agent's work

Reading the inputs

The AI agent read the instructions file, then every page it lists, then the note, and wrote down what it took from each: the control network and the isolated test network, the rule that a control address is never used for traffic, the roles of SUT and link partners, that nuttcp servers start before clients, and the direction modes relative to the SUT. It found the Network Suite page of this site and chose not to read it, so that the session would be a build and not a copy.

Designing the suite

Before writing code it wrote the design down: two groups with three tests each, laid out like the tutorial suite; the shared procedure in a package of its own, since the six tests are one procedure with a protocol and a direction fixed; the configuration fields a person will see in the panel, with the protocol and the direction kept as class attributes so nobody can turn "TCP Transmit" into something else from the GUI; and the flow, which discovers the test address on every machine, starts the servers, starts all clients together, waits, and stops the servers whatever happened.

Writing the suite

The repository it wrote, as it stands at the end of the session, follows the tutorial suite's layout the way the Network Suite does, with the shared code under a package of its own:

orcraft-test-suite-network/
	test_suite_network.py
	traffic_lib/
		base_traffic_test.py      the configuration, result and base test the six tests share
		nuttcp.py                 the nuttcp executable per platform, command lines, output parsing
		flows/flow_traffic_run.py
		steps/                    clear stale nuttcp, discover test address, start servers,
		                          start clients, wait clients, stop servers
	test_group_tcp_traffic/
		test_tcp_transmit/  test_tcp_receive/  test_tcp_bidirectional/
	test_group_udp_traffic/
		test_udp_transmit/  test_udp_receive/  test_udp_bidirectional/

A test file it wrote reads like one of the tutorial suite's:

class TestTcpTransmitConfig(DirectionalTrafficConfig):
	name = OrcraftConfigField.Text(default="TCP Transmit")
	description = OrcraftConfigField.Text(
		default=(
			"Measures the TCP throughput the SUT's test port can transmit.\n"
			"A nuttcp server runs on every link partner and the SUT sends to all of them at the same time, one client per partner, for the configured duration.\n"
			"The result is the sum of what the partners received, in Mbps."
		)
	)


class TestTcpTransmit(BaseDirectionalTrafficTest):
	protocol = Protocol.TCP
	direction = Direction.TX

	def __init__(self, config: TestTcpTransmitConfig | None = None) -> None:
		super().__init__(config or TestTcpTransmitConfig())

Running and reporting

To run it, the AI agent wrote what the Configuring a run page describes, in the run folder and not in the repository: a configured copy of every test, group and the suite with the machines, the package folder and the duration filled in, and a batch file pointing at the repository. Then it ran the batch and read the log, and each run taught it something about its own code:

  • Run 1: every test errored with a remote deadline. The AI agent found that a remote step's deadline is 30 seconds unless the step declares a duration field, gave its wait step that field, and made stuck nuttcp clients give up quickly.
  • Run 2: the servers from run 1 were still holding their ports. The AI agent added a step that clears leftover nuttcp on every machine at the start of the flow.
  • Run 3 and 4: clean failures with real numbers. Two directions measured 2372 Mbps TCP and 1000 Mbps UDP, and the other two failed with a connection timeout in every test.

Its report ended with a diagnosis and a request for the person: "traffic to GMG-TEST-2 and from GMG-TEST-3 does not pass, while the other directions reach 2372 Mbps TCP and 1000 Mbps UDP. This is almost certainly a host firewall on the test-network interface of those machines; open the nuttcp ports there." The diagnosis was half right, as the next section shows, and the numbers were real.

Opening it in the controller

Everything the AI agent wrote is an ordinary suite, so the next step is the one the AI Agents overview promises: the person adds it to the controller and looks at it. The tree shows the two groups and six tests, and a test's run panel shows the fields the AI agent chose, the endpoint fields, the package folder, the duration, the test network, the base port, the minimum throughput and, on the UDP tests, the rate:

Configured at suite scope with the same machines and run from the controller, the suite fails exactly as it did for the AI agent, six tests out of six:

The Details dialog of the bidirectional test shows the AI agent's flow step by step, the clear, discover, start and stop steps on every machine passing, and the three Wait For Clients steps failing after about 25 seconds, which is the fast failure the AI agent built in after its first run:

The person knows what the AI agent could not: the lab's firewalls open TCP and UDP 5100 to 5201 for nuttcp on every machine, and the AI agent's suite put its servers on 5000 and up. That is a one-line request.

The fix request

The request is one sentence, and it is knowledge about the lab rather than about code:

The failing directions are not the network. The lab's firewalls allow nuttcp only on TCP and UDP
ports 5100 to 5201, on every machine; port 5000 is also taken on Windows. Use ports in that range
and run the suite again.

The AI agent corrected its own earlier reading, moved the default base port to 5100 in the suite repository rather than in the run file, which does not set the port, and wrote the port plan next to the field so that a bidirectional run stays inside the allowed range.

Its run then passed, six tests out of six. Reading the log, the AI agent also noticed that every test had passed with a warning, traced it to its own Stop Servers step flagging the normal exit of a one-shot server, fixed that too, and ran once more to confirm the warnings were gone. Reloaded in the controller and run again from there, the suite passes the same way:

The person reviewing the numbers finds one that cannot be true. The bidirectional test reports the SUT receiving 3378 Mbps on a 2.5 Gbps port, and the dialog above shows why: the wait on GMG-TEST-2 took its full 10 seconds after the wait on GMG-TEST-1 had already taken 10 seconds, so that partner's stream to the SUT ran alone after the transmit streams were over, at the full rate of the port, and was then summed with them. The AI agent's "all streams concurrent" was not yet true. That, and a field the suite is missing, become the next request.

From here on

The next request is a development request, and it reads like a code review because that is what it is:

TCP Bidirectional reports SUT RX 3378 Mbps on a 2.5 Gbps port, and the wait on GMG-TEST-2 takes a
full 10 seconds after the wait on GMG-TEST-1: the receive stream from GMG-TEST-2 runs alone after
the transmit streams are over. A bidirectional test must run all its streams at the same time; find
out why that stream starts late and fix it. Also, the UDP tests report loss but never judge it: add
a Max Loss Percent field to the UDP tests, default 1, and fail a case whose worst stream loses more.
Run the suite again and report.

The AI agent took it the same way it took the first two: it read its own logs for the timing of every stream, found that the Windows partner, acting as a server and a client at the same time, ran its two directions one after the other, restructured the flow so that no machine is ever both, added the field, ran the batch and reported: every test passing, the bidirectional figures now under the port's capacity, and the UDP tests judging their loss. The person reloads the suite in the controller, sees the new field in the UDP tests' panels, runs it, and reads the numbers again. That loop is the whole method. A getting started request creates the suite; every later request, whether it corrects a fact about the lab, questions a number, or asks for a field, a case or a group, grows the same repository, and the person keeps the controller as the place where the result is looked at.

What the person keeps from this session is a suite of two groups and six tests that measures the right thing on the right network, a journal that says why every decision was taken, and a list of what to review: the checkpoints, the minimum throughput still set to report only, and the UDP rate that caps the UDP figures. What the person brought was the request, one fact about the firewalls, and an eye for a number that cannot be true.

Type to search the docs.

    ↑↓ navigate ↵ open Esc close