Network Suite

The tutorial suite runs one program on one machine at a time. A network test is a different shape of work: several machines have to act together, at the same moment, on a link that is not the one the controller uses to talk to them. The Network Performance example suite is a small suite of that kind. It measures TCP and UDP throughput between a system under test (the SUT) and its link partners with nuttcp, a network measurement tool, with the SUT transmitting, receiving, or both at once.

The suite is a separate repository laid out exactly like the tutorial suite. Clone it next to the tutorial suite and add its entry point, test_suite_network.py, to the controller the same way the Getting started guide adds the tutorial suite:

Beyond what the tutorial suite shows, this one demonstrates:

• A multi-select endpoint field that drives concurrent work: the link partners are chosen as a set, and every test opens one traffic stream per partner, all running at the same time.
• Discovery on the endpoint instead of configuration: each machine finds its own address on the test network when the test runs, so no address is typed into the controller.
• A third-party tool shipped as a package: the nuttcp binaries for every operating system travel to the endpoints in one package folder.
• Measurements aggregated across machines: the SUT's throughput is the sum over its partners, with the per-partner breakdown kept next to it.

Test setup

The setup is described in a short technical note that ships in the repository as docs/Orcraft_Network_Test_Setup.pdf; the diagrams on this page are its. The setup has two networks. The control network is the site LAN: the controller reaches every endpoint over it for registration, package uploads and step execution. The test network is a separate switch with its own subnet, 192.168.0.0/24, and only the endpoints' second Ethernet ports are plugged into it. All measured traffic flows there and nowhere else, so control traffic never disturbs a measurement and a measurement can never break control:

Every endpoint therefore has two addresses. The control address is the one the endpoint is registered with in the controller, a host name or a LAN address, and it is never used for traffic. The test address is static, has no gateway, and is not known to the controller at all:

The machines of this page's run are the lab's fleet:

The roles are chosen per run. Because the endpoints run on different platforms, the same suite is run once with each machine as the SUT, and the others as its partners, which gives a per-platform picture of every test port:

The nuttcp package

nuttcp is a single executable, built separately for each operating system. The suite expects one folder that holds the build for every platform of the fleet, and ships it to the endpoints as a package folder:

NUTTCP/bin/
	windows/nuttcp.exe
	linux/nuttcp
	macos/nuttcp

The Nuttcp Package Folder field is a PackageFolder, so the folder is uploaded to every endpoint a test uses and each step gets the path where it landed on its own machine. The step then picks the binary for the operating system it is running on, and gives an uploaded Unix binary its execute bit back, which package uploads do not keep. The default points at C:\Orcraft\apps\NUTTCP\bin; browse to your own copy if it lives elsewhere.

The one thing to prepare on the machines is the firewall. The receiver of a stream runs a one-shot nuttcp server on a control port and a data port, and both must be open for TCP and UDP:

Opening the range 5100 to 5201 on every machine covers both roles, whichever machine is the SUT of a run.

Suite structure

The repository follows the tutorial suite's layout, with two groups of three tests that share one flow:

orcraft-test-suite-network/
	test_suite_network.py
	common/
		nuttcp_app.py            the nuttcp executable: server, client, output parsing
		test_network.py          the test network and the lookup of a machine's address on it
		traffic_test_config.py   the configuration every traffic test shares
		flows/flow_run_traffic.py
		steps/                   verify nuttcp, find test IP, start server, start client,
		                         wait client, stop server
	test_group_tcp/
		test_tcp_tx/  test_tcp_rx/  test_tcp_bx/
	test_group_udp/
		test_udp_tx/  test_udp_rx/  test_udp_bx/

Shared configuration

Source file:

Every traffic test takes the same fields, declared once in a base configuration class. The SUT is a single Endpoint; the link partners are a MultiSelectEndpoints, the field type from Endpoints Config Types that holds a set of machines:

class TrafficTestConfig(OrcraftTestConfig):
	sut_endpoint = OrcraftConfigField.Endpoint(default="localhost", label="SUT Endpoint")
	link_partner_endpoints = OrcraftConfigField.MultiSelectEndpoints(
		default=("localhost",),
		label="Link Partner Endpoints",
	)

	nuttcp_package_folder = OrcraftConfigField.PackageFolder(
		default=r"C:\Orcraft\apps\NUTTCP\bin",
		label="Nuttcp Package Folder",
		required=True,
		layout=OrcraftConfigField.Layout.NewRow,
	)

	duration_sec = OrcraftConfigField.Numeric(
		default=10,
		label="Duration Sec",
		layout=OrcraftConfigField.Layout.NewRow,
	)
	min_throughput_mbps = OrcraftConfigField.Numeric(default=100, label="Min Throughput Mbps")

There is no field for the test addresses. The address a machine has on the test network is a fact about that machine, so the suite asks the machine:

# The network under test. Every machine of the setup has one address on it, separate from
# the address the controller uses to reach the machine, and the tests find that address by
# themselves: nothing about it is configured in the controller.
TEST_NETWORK = "192.168.0.0/24"


def find_test_ip(network: str = TEST_NETWORK) -> str:
	"""The IPv4 address this machine has on the given network, or empty when it has none."""
	wanted = ipaddress.ip_network(network, strict=False)
	for candidate in _local_ipv4_addresses(wanted):
		try:
			if ipaddress.ip_address(candidate) in wanted:
				return candidate
		except ValueError:
			continue
	return ""

The Find Test IP step runs this on the SUT and on every partner before any traffic, and fails with a plain message when a machine has no address on the network, which is how a cable left unplugged shows up.

One stream per partner

Source file:

The flow turns the direction and the partner set into streams. A stream is one nuttcp transfer between the SUT and one partner: the receiver runs a one-shot server, the transmitter runs the client. With the SUT transmitting, the servers sit on the partners; with the SUT receiving, every partner gets its own server on the SUT, on its own ports:

def _build_streams(self, partners: tuple, *, sut_ip: str, partner_ips: list[str]) -> list[TrafficStream]:
	direction = str(self.config.direction).upper()

	streams: list[TrafficStream] = []
	for index, (partner, partner_ip) in enumerate(zip(partners, partner_ips)):
		name = endpoint_name(partner)

		# The SUT transmits: the server sits on the partner, one per machine, on the partner port
		if direction in {"TX", "BX"}:
			streams.append(
				TrafficStream(
					direction="TX",
					partner=name,
					transmitter=self.config.sut_endpoint,
					receiver=partner,
					target_ip=partner_ip,
					control_port=LINK_PARTNER_CONTROL_PORT,
				)
			)

		# The SUT receives: every partner gets its own server on the SUT, on its own ports
		if direction in {"RX", "BX"}:
			streams.append(
				TrafficStream(
					direction="RX",
					partner=name,
					transmitter=partner,
					receiver=self.config.sut_endpoint,
					target_ip=sut_ip,
					control_port=sut_control_port(index),
				)
			)

	return streams

The streams then run together. The flow starts a server for every stream, starts a client for every stream, and only then waits for the clients one by one. A client is started in a background thread on its machine and its task is carried on the step result, the pattern from Handling Asynchronous Steps, so starting the next client does not wait for the previous one to finish. With three partners the SUT carries three streams at once, and a bidirectional test carries six.

Everything is measured at the SUT's port. A UDP rate limit is the load offered at that port, so with several partners each stream gets an equal share of it, 1 Gbps to three partners becoming three streams of 333 Mbps; without the split, two partners at 2 Gbps would offer 4 Gbps to a 2.5 Gbps port and the loss would measure the overload, not the port. When the clients are done, the flow sums each direction over the partners into the SUT's numbers and keeps the per-partner breakdown as text. The pass criteria stay per stream: each partner has to reach the minimum throughput by itself, and for UDP keep its loss within the limit.

Traffic directions

Directions are relative to the SUT. TX means the SUT transmits to every partner, RX means every partner transmits to the SUT, and BX runs both at the same time:

Each group has one test per direction. The TCP tests sweep nuttcp's buffer length and socket window size; the UDP tests sweep the datagram length and the transmit rate, and check the datagram loss as well as the throughput:

Running the suite

Add the suite to a session that has the fleet's endpoints registered. Every test comes up with localhost in both endpoint fields and the default package folder, so a reader with a single machine sees the panels at once, but a run needs the real machines:

The endpoints are the same for every test, so they are set once at suite scope. Select the suite, click Configure, set SUT Endpoint to GMG-TEST-1 and tick GMG-TEST-2, GMG-TEST-3 and GMG-TEST-5 under Link Partner Endpoints on one test, then apply each column to all tests from its header. The table shows the whole suite configured, and the page of the selected test below it:

Selecting a test shows its own run panel with the endpoints it will use and its cases:

Click Run on the suite panel and confirm. The tests queue up and run one after the other, each locking the SUT and the three partners for its duration.

The run in the log

Each test starts by locking its four machines and uploading the suite and the nuttcp package to each of them in parallel. The flow then verifies nuttcp on every machine and asks each one for its test address:

                                     Step: Find Test IP | Configuration            

Parameter       | Value
-----------------------------------------------------------
Start Time      | 2026-09-16  14:06:26.057
Target Endpoint | GMG-TEST-1:50051 [Secure] (v0.1.0.dev218)
Network         | 192.168.0.0/24

Test IP on 192.168.0.0/24: 192.168.0.11

     Checkpoint Pass   This endpoint is on 192.168.0.0/24 as 192.168.0.11.


                              Step: Find Test IP | Result      

Parameter    | Value
---------------------------------------
Name         | Step: Find Test IP
Status       | PASS
Finish Time  | 2026-09-16  14:06:26.093
Elapsed Time | 0.036 s

The servers start, the clients start, and every client's wait step reports the nuttcp command it ran and the summary line nuttcp prints when the transfer ends, with the throughput, the CPU load on both sides and, for TCP, the round-trip time:

                                  Step: Wait Nuttcp Client | Configuration         

Parameter       | Value
-----------------------------------------------------------
Start Time      | 2026-09-16  14:06:30.223
Target Endpoint | GMG-TEST-1:50051 [Secure] (v0.1.0.dev218)
Duration Sec    | 10

nuttcp client command: nuttcp.exe -T10 -P5200 -p5201 -l64K -w256K 192.168.0.12
1024.9375 MB /  10.02 sec =  858.4523 Mbps 3 %TX 20 %RX 0.62 msRTT

     Checkpoint Pass   nuttcp client completed: 1024.9375 MB /  10.02 sec =  858.4523 Mbps 3 %TX 20 %RX 0.62 msRTT

The flow ends with one checkpoint per stream and the result table, where the SUT's numbers are the sum over its partners and the breakdown sits next to them. Here the SUT's port carried its full 2.5 Gbps while feeding three partners at once:

     Checkpoint Pass   SUT TX to GMG-TEST-2 throughput 858.5 Mbps reaches the minimum of 100.0 Mbps.


     Checkpoint Pass   SUT TX to GMG-TEST-3 throughput 859.5 Mbps reaches the minimum of 100.0 Mbps.


     Checkpoint Pass   SUT TX to GMG-TEST-5 throughput 713.7 Mbps reaches the minimum of 100.0 Mbps.


                                                Flow: Run Traffic | Result                           

Parameter               | Value
---------------------------------------------------------------------------------
Name                    | Flow: Run Traffic
Status                  | PASS
Finish Time             | 2026-09-16  14:06:40.362
Elapsed Time            | 16.419 s
SUT TX Throughput Mbps  | 2431.7
SUT TX Transferred MB   | 2902.9
SUT TX Retransmissions  | -
SUT TX Loss Percent     | -
SUT TX Per Partner Mbps | GMG-TEST-2: 858.5; GMG-TEST-3: 859.5; GMG-TEST-5: 713.7
SUT RX Throughput Mbps  | -
SUT RX Transferred MB   | -
SUT RX Retransmissions  | -
SUT RX Loss Percent     | -
SUT RX Per Partner Mbps | -

Results

When the suite finishes, the tree carries a run row under every test with its endpoints and status, and the group and suite rows sum their cases:

The Details dialog of a test lists every step with the machine it ran on, which is where the concurrency of the flow becomes visible: the servers on three machines, the clients on the fourth, and the waits going back to each client's machine. The dialog is too wide for one picture, so here it is in two halves. The left half holds the description, the configuration, the cases and the run tree with the endpoint of every step:

The right half continues the same rows with the result columns. Every result field of the flow and of the steps is a column of its own: the flow rows carry the SUT's TX and RX throughput, transferred megabytes, retransmissions and the per-partner breakdown, the Verify Nuttcp rows the binary's place in the package and the nuttcp version, the Find Test IP rows the test addresses, the Start Nuttcp Server rows the server PIDs, and the Wait Nuttcp Client rows each client's throughput, duration, retransmissions and CPU load:

Numbers in a dialog are a start, not an analysis. The two buttons at the bottom take the whole scope out: Export to XLSX writes a workbook with the summary, the configurations and this results tree, every column included, for pivoting the throughput by partner, direction and window size (see Export to XLSX), and Export for Agents writes the summary and the run logs as plain text so an AI agent can read the whole run and explain a slow partner or a lossy case (see Export for Agents).

Reading a failure

Two UDP tests of this run failed, and the suite says exactly where. In both, the streams from GMG-TEST-2 and GMG-TEST-3 completed, and the one from GMG-TEST-5 to the SUT did not: the nuttcp client on the macOS machine sent for the whole duration and then reported that nothing had arrived, while the SUT's server for that partner was still waiting:

nuttcp client command: nuttcp -T10 -P5104 -p5105 -u -l1K -w4M -R1G 192.168.0.11
                        nuttcp-t: v8.1.4: Error: IO: Operation timed out
                        errno=60

     Checkpoint Fail   nuttcp client did not complete.

                            ### Checkpoint Failure Details ###

                             nuttcp-t: v8.1.4: Error: IO: Operation timed out
                             errno=60

The same machine receives UDP without loss in the UDP Transmit test and carries TCP in both directions, so this is the test port of that one machine failing to deliver UDP, not the link and not the suite. That is the kind of finding the suite exists for. Select the UDP Traffic group, take GMG-TEST-5 out of its link partners on the group panel, and run the group. While it runs, the tree shows the running test with its progress bars, the queued one behind it, and the machines it holds marked busy in the Endpoints strip:

With the two partners whose ports do send UDP, the group passes in full:

The next page, FTDI Suite, moves from the network to the bench: hardware on a breadboard, driven from two machines over a handful of wires.