FAQ

Orcraft is the platform for test development and orchestration that lets you and your AI agents run the entire testing operation. A plain-Python test framework gives suites a defined structure. A controller runs those suites from a GUI, the command line or a CI pipeline and collects the logs and results in one place. An endpoint application on each test machine lets the controller run tests there, so a run covers the local machine or a fleet of remote machines with the same suites.

Your AI agents work across all of them. They write suites in the framework's structure, run them on the machines registered in the controller and read the logs the runs leave behind. The person decides what to test, reviews what your AI agents wrote in the controller GUI, runs it and corrects it, unless the whole loop is handed to your AI agents and runs fully automated.

Validation and engineering teams that test software, firmware or hardware on real machines: lab setups, devices under test, cross-platform automation and CI execution. If your tests need to run on more than one machine, be configured by people who did not write them, or produce results someone has to report on, Orcraft is built for you. If you want your AI agent to do the developing, running and reporting, and keep the GUI for review, Orcraft is built for that too.

No. A test automation system executes scripts that someone wrote for it and schedules them. Orcraft covers the testing operation as a whole: the suites are developed continuously, so a fixed bug or a new feature becomes a test; every suite has the same structure, so the controller derives the GUI, the run plan and the reports from the code; a run reaches every machine in the lab through the endpoints; and the results of every run stay in one place.

The difference shows with AI agents. An automation system gives them a trigger to pull. Orcraft gives them a structure to write into and a lab to run on, so they can develop the tests, pick the machines, run and read the results, while the person reviews the work in the controller GUI.

The code comes first and everything else follows from it. Suites, groups, tests, flows and steps are written directly in Python, and the controller reads that code to derive the configuration widgets in the GUI, the run plan, the CLI defaults and the CI snapshot. The suite is the single source of truth, so the GUI, the command line and CI can never drift apart.

You write a test suite once and re-run it under different settings without touching the code. A suite developed against localhost runs on remote endpoints by selecting them in the run panel. A parameter added to the configuration becomes a GUI field that anyone can change. Dimensions expand the same test into a different case matrix per run. The code stays the same; only the run configuration changes. Your AI agent configures a run the same way without the panel: it redeclares the fields in a subclass, runs it and reads the log.

There is no second layer beside the code: no separate configuration language for your tests, and no plugins or scripts to keep in sync with them. Test configuration, including the widgets shown in the GUI, is declared in Python next to the test that uses it, and the controller generates the interface from that declaration at run time. If it is not in the Python, it does not exist.

The controller does keep save files of its own, in the way an editor keeps a workspace file. A session is stored as a JSON file that you reopen, share or re-run, never write or edit by hand. A CI snapshot is exported as Python. Neither is something you maintain; both are produced from the code.

No. Orcraft has no DSL (domain-specific language) of its own. Tests are ordinary Python classes with a few base classes and field types imported from the orcraft package. Anything Python can do, a test step can do, including calling the libraries and tools your team already uses. Orcraft runs on Python 3.14 or newer.

Python is the native language of Orcraft: the framework is a Python library and tests are written as Python classes. That is where Python ends. A step is ordinary Python code, so it can launch an executable, drive a command-line tool, talk to firmware over a serial port, call a REST API, reach a device over the network or use a vendor SDK. The system under test can be written in any language, or not be software at all. Python is the wrapper that gives it structure, configuration and results.

No. Orcraft is built for validation labs, where the thing under test is as often a device, a board, an instrument or a complete system as it is a program. An endpoint is any machine that can reach the hardware: a lab PC wired to a device under test, a server on the same network as the instruments, a machine with the vendor's tools installed. Steps run there and talk to the hardware over whatever interface it offers. Software, firmware and hardware validation all use the same suites, the same GUI and the same results.

No. Orcraft has no built-in AI, no AI provider to configure and no API key to enter. An AI agent is one way to develop and run suites: it works through the same framework and the same files as a person, and everything it writes opens in the controller GUI. Without one, the GUI, the CLI and CI work exactly the same.

Any AI agent that can read a folder, write Python and run a command on the machine where the suites are cloned, such as Claude Code, OpenAI Codex or Gemini CLI. Nothing in Orcraft is specific to one AI agent: its inputs are a suite repository, the endpoints file saved from the controller and this documentation, and its output is ordinary Python files and a log. The Inputs page has a template for the instructions file your AI agent reads at the start of every request.

The person decides what to test and asks for it. Your AI agent writes the suite, runs it on the machines named and reads the log. Everything it writes is an ordinary suite, so the person opens it in the controller GUI, reviews it, runs it and corrects what needs correcting, then hands it back to your AI agent or to CI. Standing instructions, such as adding tests as the system changes or turning every fixed bug into a test, are given once and reviewed the same way, one commit at a time.

Through the same endpoint application the controller uses, and only the machines the team registered. The controller's Endpoints panel saves the registered machines to a file, and that file is the only list of machines your AI agent works from. A run started by your AI agent uses the same authorization token and certificates as the controller, as described in Running a batch, so an endpoint that accepts the controller accepts your AI agent and no other.

It could, as it can edit any file, so the page it reads forbids it. Its rules say: never edit a test to run it. To run a test your AI agent writes a separate file that subclasses it and redeclares the fields it needs, the same mechanism a CI snapshot uses, and keeps that file outside the suite repository. When the request is to change a test, your AI agent edits the repository like a developer would, and the change is reviewed in the GUI or in the commit.

Windows, Linux and macOS, for both the controller and the endpoints, and they can be mixed freely in one session. Every release is built for Windows x64, Linux x86_64, Linux ARM64 and macOS on Apple silicon and Intel. The controller GUI needs a desktop session. Endpoints have no GUI and run headless, on a server or over SSH. The validated versions are listed in the System reference.

No. The controller runs suites on localhost out of the box, so you can develop and run everything on one machine. Endpoints are configuration, not code: a suite developed against localhost becomes a distributed suite by selecting endpoints in the run panel, with no change to the test body.

Test traffic, logs and results stay between your controller and your endpoints. The controller makes two optional outbound calls, a version check against PyPI at startup and a status check for suites kept in Git, and both fail quietly when the network is unavailable. The controller contacts no AI provider. Isolated machines can install from a copied wheel file.

Orcraft™ is a commercial product of GM Core Ltd. and its source code is proprietary. It installs from PyPI with pip install orcraft. The Licensing page explains how endpoint machines are licensed, and the pricing page has the plans and prices.

Hell yeah! Every component of Orcraft (the controller GUI, the framework, the endpoint application, the Maze app, the Tutorial suite and so on) is validated with Orcraft suites, written like any suite you would write. The tests are developed and enhanced continuously, and they run continuously.

We see this as the strongest argument for the platform: if Orcraft can validate a system as complex as Orcraft, it can validate what you build.

An AI agent can already write Python, read a log and run a command. What it lacks in a testing lab is a structure to write tests in, a list of the machines it may use and a way to run on them. Orcraft supplies all of that without an API to learn: a suite is plain Python with its configuration declared as fields, the controller's endpoints file is the list of machines, and a run is a small file your AI agent writes and executes.

With that in place the lab runs continuously. Your AI agent creates the first suite from the documentation of the system under test. As the system changes, it compares what the system does with what the suite checks and adds the missing tests. Every bug that gets fixed becomes a test that reports the bug's return. Every merge runs the suite from the pipeline on the lab machines. The AI Agents section shows the requests and the standing instructions that set this up.

The person stays in charge. Everything your AI agent writes is an ordinary suite that opens in the controller GUI, where it is reviewed, run and corrected like any suite written by hand.

The same Python suite runs from the controller GUI, from the command line, from a CI pipeline and from your AI agent. A run confirmed in the GUI is exported as a snapshot that reproduces it exactly elsewhere, so there is no "works in the GUI but not in CI". Your AI agent runs the same suite by writing a file that sets the fields it needs, and reads the log afterwards.

Because the GUI is generated from the code, a lab technician or a tester who never opens the Python can select endpoints, set parameters, filter a matrix and run the suite. Whoever writes the suite, your AI agent or a developer, decides what is configurable by declaring it; everyone else gets a form. Sessions and results are shared the same way.

The controller uploads the test package to each endpoint at run time and updates the endpoint software whenever versions differ. Endpoints are registered once and can be shared between controllers and across the team.

Dimensions expand one test into a planned matrix of cases, filtered from the GUI and executed in defined or randomized order over configurable cycles. The same test can also fan out to several endpoints, one independent run each, in parallel.

Live logs from every endpoint, step-level statuses, statistics that roll up automatically from case to suite, spreadsheet export and portable session files. Sessions are saved, reopened later and re-executed with the same plan. Any scope can also be exported for AI agents: a plain-text summary of its results with the logs of its runs, in one folder for your AI agent to analyze.

Every level of the hierarchy is optional. A single test file with a main() runs on its own, on localhost, with no group, suite or endpoint around it. The same file later joins a suite that runs across a lab full of endpoints, with nothing rewritten. There is no infrastructure to stand up before the first test runs.

Windows, Linux and macOS, as controller or endpoint, in any combination within one session. Endpoints have no GUI and run headless on a server or over SSH. Everything installs with pip, so the toolchain is the Python, editor and Git your team already has.

Controller-to-endpoint communication can be secured with mutual TLS and token-based access control, so endpoints in remote labs or across an external network can join a session safely. The controller makes outbound connections only and needs no inbound firewall rule.

A suite can be kept in a Git repository or supplied as an archive. For Git-backed suites the controller checks the remote in the background and offers a fast-forward pull, without ever blocking a run.

pytest and unittest run test functions on one machine and report pass or fail. Orcraft covers what happens around that: a structured hierarchy of suites, groups, tests, cases, flows and steps, a GUI generated from the code so anyone can configure a run, dispatch of individual steps to remote endpoints, parametrized case matrices, and results that roll up from step to suite. Existing Python test logic can be called from an Orcraft step, so you do not have to throw anything away.

See the note on other products

Robot Framework describes tests in a keyword language backed by Python libraries, so a team maintains both the keyword files and the code behind them. In Orcraft the test is the Python code, and the controller derives the configuration GUI, the run plan and the reports from it. There is no second layer to keep aligned, and no resource files to distribute to the machines that run the tests.

See the note on other products

Not directly. Those tools track test cases, plans and results that are executed elsewhere. Orcraft is the execution side: it runs the tests, keeps every session with its live logs and step-level results, and exports structured spreadsheets and portable session files. Many teams find the saved sessions and exports cover their reporting needs; others keep their management tool and feed it from the exports.

See the note on other products

No, it runs inside them. A CI runner is a controller installation running in CLI mode, using a snapshot exported from the controller where the run was configured and confirmed in the GUI. The pipeline triggers the run; Orcraft handles distributing the package to endpoints, executing the plan and collecting results. You write no per-test pipeline glue.

See the note on other products

Both run sequences of steps against hardware and report on them. In NI TestStand the sequence is built in the tool's own editor, saved in its own file format and run on Windows only, and the step code lives in separate modules that the sequence calls. In Orcraft the sequence and the steps are one Python suite. There is no editor to learn and no file format to maintain: the controller derives the GUI from the code, an AI agent can write the suite because it is plain Python, and the controller runs it on Windows, Linux and macOS, dispatches steps to remote machines over the network and installs with pip.

See the note on other products

OpenTAP is Keysight's open-source test sequencer. Its engine is C# on .NET, a test plan is a sequence of steps saved as a .TapPlan XML file, built in the tool's editor or a text user interface and run with the tap run command, and the steps come from plugins written in C# or Python. It runs on Windows and Linux. That is two layers to keep aligned, the plan file and the plugins behind it. In Orcraft the plan and the steps are one Python suite: an AI agent can write it, the controller derives the GUI from it, runs it on Windows, Linux and macOS, dispatches steps to remote machines over the network and installs with pip, with no plugin toolchain and no .NET.

See the note on other products

labgrid is an open-source Python library for controlling embedded boards in a lab. Exporters expose the boards connected to each lab host, a coordinator manages access to them, and a client on the test host takes a board over the network and drives its console, power and bootloader. It is not a test framework: tests are written for pytest with the labgrid plugin, and they run on the test host, so the GUI, the run plan and the roll-up of results are whatever pytest and its plugins provide. In Orcraft the test steps themselves run on the endpoints, any Windows, Linux or macOS machine with the endpoint application installed with pip, and the suite structure gives the run a configuration GUI, a plan, live logs and results rolled up from step to suite, on the local machine and in CI alike.

See the note on other products

LAVA is Linaro's open-source continuous integration system for boards. A job is a YAML file submitted to the LAVA server; a dispatcher on a lab host deploys an operating system image onto a board, boots it and runs the tests on it, and the server tracks the results. The job describes deploy, boot and test for one device type, and the system needs a server and dispatchers installed and administered. An Orcraft test is Python code that runs on any registered machine, the controller and the endpoints install with pip, and the same suite runs from the controller's GUI, from the command line and in a CI pipeline, with the results and logs of every run in the controller.

See the note on other products

OpenHTF is Google's open-source Python framework for hardware testing on a test station. A test is a sequence of phases, plain Python functions; plugs wrap the instruments and the device under test, measurements with validators decide the outcome, and an operator web UI shows the station. The test runs on the station it was written for, and running it elsewhere, configuring it or collecting results across stations is left to the team. In Orcraft the suite is structured into groups, tests, cases, flows and steps, and the controller runs the steps on the machines registered as endpoints, from the configuration GUI it derives from the code, from the command line or in CI, and rolls the results up from step to suite in one place.

See the note on other products

In-house frameworks tend to grow a config format, a copy-to-lab-machine procedure, a results spreadsheet and a way to run in CI, each maintained by hand. Orcraft supplies all of that from the Python suite itself, so the team's effort goes into the tests. Because steps are plain Python, the scripts you have can usually be wrapped rather than rewritten.

An AI agent can write a script for anything, and a lab full of scripts is what teams end up maintaining. Orcraft gives your AI agent a structure to write into, so every suite it produces has the same shape, the same configuration fields and the same results, whoever asked for it and whenever. That structure is also what the controller reads: a suite written by an AI agent gets a GUI, endpoint selection, a case matrix, live logs and roll-up statistics without any of that being written. A script gets none of it.

Minutes. Install Python, run pip install orcraft, start the controller and run the Hello World suite from the Tutorial on localhost. Remote endpoints, matrices and CI come later, one chapter at a time.

Install Orcraft, register the lab machines in the controller and save the endpoints file from the Endpoints panel. Clone the suite repositories, or start with none, into the folder your AI agent works from, and put the instructions file from the Inputs page beside them. Then make the first request: create a suite from the documentation of the system under test, or run an existing test on the machines you name. The Example prompts page shows what a request looks like.

It depends on whether you want a secure connection. An insecure endpoint takes a few minutes: install Python, run pip install orcraft, start the endpoint with security disabled and add it in the controller by name or IP address. It is the fastest way to try remote execution on a trusted network.

A secure endpoint adds the security setup procedure in the controller's Security tab, which generates the authorization token and the certificates for mutual TLS, and a firewall rule on the endpoint. Plan on about 15 minutes for the first one. An endpoint added without security can be switched to secure later by editing it in the controller, so you can start fast and harden afterwards.

Yes. The Demo page shows the controller in action, and the Tutorial walks through developing, configuring, running and analyzing a suite with screenshots.

Write to support@orcraft.app or use the Contact page. We answer every message.