Packages Uploading

A step that runs on an endpoint needs two things on that machine: the suite's code, so the step can be executed there at all, and whatever files the step works on. Orcraft moves both for you before the run, and calls what it moves a package. The suite itself is the test package, uploaded as a whole. A file or folder that a test needs is declared as a package field, a configuration field whose value is uploaded along with the code and rewritten, inside the step, to the path where it landed.

Uploading is governed by policies, so a run is neither slowed by re-sending what a machine already has nor broken by a machine that misses something:

• Always uploads before every run. It is the safe choice while the suite or the files are still changing.
• If Missing uploads only when the endpoint does not already hold an identical copy. This is the default, and the usual choice once things are stable.
• Never uploads nothing and expects the endpoint to have the files already.

The endpoint application is a package of a kind too. Its version on each machine is compared with the controller's, and the Update Endpoint App row decides whether it is brought up to date before the run: On Mismatch by default, Always, or Never. This is how a whole lab follows a controller upgrade without anyone logging in to the machines.

This page follows the uploads of two tests of the Endpoints group. The first, Default Endpoint and Package Upload from the previous page, shows the test package and one package folder arriving on a machine. The second, Package Upload Parameter Types, declares every package field type and binds each to the endpoints that need it.

The test package

Source directory:

Select Default Endpoint and Package Upload with Test Endpoint still set to the Linux machine. Set Upload Test Package to Always, and the Upload choice next to the Maze Executable Folder to Always as well, so both uploads are visible on every run. Click Update Plan, then Run and confirm. Search the log for package:

The run starts by taking the endpoint for itself, the Lock acquired line, and then uploads. The first progress bar sends orcraft-test-suite-tutorial, the suite's code, and the second sends v3.0.0, the Maze Executable Folder. Each upload ends with an Uploaded and activated line. With If Missing the same lines would read Existing remote package already active on the second run, because the endpoint keeps what it received and recognizes an identical copy by its content.

The suite is uploaded as a whole, not test by test, so a step can import the suite's shared modules on the endpoint exactly as it does on the controller. The Maze folder is uploaded because the field that names it is a PackageFolder, not a plain Folder; a plain folder field would hand the step a path that exists only on the controller.

Package field types

Source directory:

There is a package field for each shape a test's inputs can take: a single file, a single folder, a list of files and a list of folders. Each is the uploading counterpart of a filesystem field from the Basic Configuration Parameters group, with the same Browse dialog and the same value type:

• PackageFile uploads one file.
• PackageFolder uploads one folder with everything in it.
• PackageFileList uploads any number of files.
• PackageFolderList uploads any number of folders.

Without further instruction a package field is uploaded to every endpoint the test uses. That is fine for a small binary and wasteful for a large data set that only one machine reads, so a field can be bound to the endpoint fields that actually need it with the endpoints option. Package Upload Parameter Types declares four endpoint fields, one of each kind seen on the previous page, and one package field of each type, each bound differently:

test_endpoint = OrcraftConfigField.Endpoint(default="localhost", label="Test Endpoint")
endpoint_1 = OrcraftConfigField.EndpointFanout(default=("localhost",), label="Endpoint 1")
endpoint_2 = OrcraftConfigField.Endpoint(default="localhost", label="Endpoint 2")
aux_endpoints = OrcraftConfigField.MultiSelectEndpoints(default=("localhost",), label="Aux Endpoints")

maze_executable_folder = OrcraftConfigField.PackageFolder(
	default="",
	label="Maze Executable Folder",
	required=True,
	layout=OrcraftConfigField.Layout.NewRow,
)

maze_file = OrcraftConfigField.PackageFile(
	default="",
	label="Maze File",
	required=True,
	layout=OrcraftConfigField.Layout.NewRow,
	endpoints=[endpoint_1],
)

maze_files = OrcraftConfigField.PackageFileList(
	default=(),
	label="Maze Files",
	required=True,
	layout=OrcraftConfigField.Layout.NewRow,
	endpoints=[endpoint_2],
)

maze_folders = OrcraftConfigField.PackageFolderList(
	default=(),
	label="Maze Folders",
	required=True,
	layout=OrcraftConfigField.Layout.NewRow,
	endpoints=[test_endpoint, aux_endpoints],
)

The Maze Executable Folder has no endpoints option and goes everywhere, because every step runs the application. The single Maze File goes only to the fan-out endpoints, the Maze Files only to Endpoint 2, and the Maze Folders to the test endpoint and the multi-select set. A binding names endpoint fields, not machines, so it holds whatever machines are chosen later.

Configuring the uploads

Select the test. Set the Windows machine as Test Endpoint, the Linux machine as Endpoint 1 and as the single Aux Endpoints entry, and the macOS machine as Endpoint 2. Then browse the package fields: the v3.0.0 folder as before, its maze.json as the Maze File, maze2.json and maze3.json as the Maze Files, and the v2.0.0 and v1.0.0 folders as the Maze Folders:

Every package field carries two things a plain filesystem field does not. The Endpoints line states the binding as declared, All for the unbound folder and the field names for the others. The Upload choice is the field's own policy, separate from the test package's: a large data set can stay on If Missing while the code is set to Always.

The uploads in the log

Click Update Plan, Run, and confirm. When the run finishes, search the log for Lock acquired:

The three machines are locked together, because the run uses all of them at once, and the uploads run on them in parallel. The test package is sent to every machine, as Always demands. The other lines read already active, because these files were uploaded by the earlier runs of this page and their fields are on If Missing. Where they appear is the binding made visible: the v3.0.0 package is checked on all three machines, and the v2.0.0, v1.0.0 package only on GMG-TEST-3 and GMG-TEST-1, the machines behind Aux Endpoints and Test Endpoint.

The maze files have no line of their own. Package fields whose files lie in the same folder are uploaded together as one package, and the three maze files live inside the v3.0.0 executable folder, so they travel with it. Their bindings still hold: the package sent to GMG-TEST-5 carries maze2.json and maze3.json but not maze.json, which only GMG-TEST-3 receives.

Using uploaded paths

Inside a step the package field holds the path on the machine the step is running on, not the path browsed on the controller. The step that solves the single uploaded file therefore checks the file exists and uses it directly:

def _run_step(self) -> OrcraftTestStepResult:
	self.result = StepSolveUploadedFileResult()

	maze_file = normalize_long_path(self.config.maze_file)

	self.checkpoint(
		bool(maze_file) and os.path.exists(maze_file),
		f"Uploaded maze file exists: {maze_file}",
		f"Uploaded maze file is missing: {maze_file}",
	)

	if not (maze_file and os.path.exists(maze_file)):
		return self.result

	result = MazeApp(
		normalize_long_path(self.config.maze_executable_folder),
		"solve",
		maze_file=maze_file,
	).run()

	self.result.path_found = result.solved

The flow sends this step to each fan-out endpoint, the file list step to Endpoint 2 and the folder list step to the test endpoint, which is exactly where the fields were bound. The folder list step runs each uploaded version's own executable, so the v2.0.0 and v1.0.0 binaries are exercised on the Windows machine with no copy of them made by hand.

The end of the log sums it up: two versions solved by the folder list step, five mazes solved in all, and the three locks released. Open the run's Details:

Each of the three steps ran on the machine its package field was bound to, and the configuration table keeps the browsed paths of every package field next to the endpoints they were sent to.

Every step so far found its endpoint waiting. The next page, Controlled Endpoints Disconnection, takes the endpoint down in the middle of a flow and waits for it to come back.