Instructions for AI agents

This page is for an AI agent that develops or runs Orcraft test suites. Orcraft is a platform for test development and orchestration. A test suite is ordinary Python written with the orcraft package, and the same files run from the controller GUI, from the CLI, from CI and from you. The person's instructions file gives the paths you work with: the suite repositories, the run folder, the endpoints file saved from the controller and the Python with the orcraft package installed.

Documentation

Read the pages below before the first request. The Technical details page describes the files you read and write; the Framework reference describes the classes and fields a suite is written with; the tutorial suite is the layout every new suite follows.

Working with Orcraft

• Endpoints file: the machines and how to address them.
• Configuring a run: the subclass that fills in the values of one run.
• Running a batch: how that file is run and what a run needs to reach remote endpoints.
• Reading the log: the tables to read a result from.

Framework reference

• Orcraft framework overview
• Orcraft Configuration Fields
• Orcraft Statuses
• Orcraft Result Fields
• Orcraft Test
• Orcraft Matrix
• Orcraft Test Step
• Orcraft Test Flow
• Orcraft Endpoints
• Orcraft Setup, Teardown and Exit
• Orcraft Test Group
• Orcraft Test Suite
• Orcraft Test Batch
• Orcraft Logger

Reference suite

• Tutorial suite: the layout every new suite follows.

Rules

• Never edit a test to run it. Configure a run with a subclass that redeclares the config fields it needs.
• Configured copies and their batch go into the run folder, never into a suite repository.
• Every batch item points import_roots at the suite repository, as the Item paths section describes.
• The endpoints file is the only list of machines. A machine named by host name, by address or by operating system resolves through its entries, and its value goes into an endpoint field. A request for any available machine goes in as an endpoint pool, and the configured copy names the endpoints file in endpoints_file_path, as Technical details shows.
• A test that imports a package outside the Python standard library adds it to the suite repository's requirements.txt; the controller installs it on the endpoints.
• A required field whose value is neither in the code nor in the instructions file is asked for, never guessed.
• If requested, run the batch from the run folder. After the run, read the log under Logs/ there and summarize the result from its tables.

Links

The full addresses of the pages linked above:

https://orcraft.app/agent-instructions/
https://orcraft.app/ai-agents/details/#endpoints-file
https://orcraft.app/ai-agents/details/#configuring-a-run
https://orcraft.app/ai-agents/details/#running-a-batch
https://orcraft.app/ai-agents/details/#reading-the-log
https://orcraft.app/framework/
https://orcraft.app/framework/configuration-fields/
https://orcraft.app/framework/statuses/
https://orcraft.app/framework/result-fields/
https://orcraft.app/framework/test/
https://orcraft.app/framework/matrix/
https://orcraft.app/framework/steps/
https://orcraft.app/framework/flows/
https://orcraft.app/framework/endpoints/
https://orcraft.app/framework/setup-teardown-exit/
https://orcraft.app/framework/groups/
https://orcraft.app/framework/suites/
https://orcraft.app/framework/batches/
https://orcraft.app/framework/logging/
https://github.com/GMCoreAI/orcraft-test-suite-tutorial