Orcraft 1.3.0 released 19-Nov-2026.

Orcraft 1.3.0 released 19-Nov-2026.

Orcraft

Run the entire lab testing operation with your AI Agents.

Watch the demo

Orcraft and other test tools

Orcraft is the platform for test development and orchestration that lets you and your AI agents run the entire testing operation. You and your AI agents write the test logic, in plain Python, and nothing else. The Orcraft framework takes care of the rest: the GUI of a run, the machines each step runs on, the logs and the results, and the snapshot that repeats a run in a CI pipeline.

The table sets Orcraft beside the test tools engineers already know, one characteristic to a row. Each name opens a page that sets Orcraft beside that tool alone.

The tools were picked to cover the kinds of tool a lab chooses between: a commercial test sequencer (NI TestStand), an open-source one (OpenTAP), a keyword-driven framework (Robot Framework), the Python test framework (pytest), a hardware test library (OpenHTF) and two systems for boards in a lab (labgrid and LAVA). If the tool you use is missing, tell us.

See the note on other products

Side by side

yes  ·  no  ·  words where a tool does it in its own way

Orcraft NI TestStand Robot Framework pytest OpenTAP OpenHTF labgrid LAVA
What it is Test framework, controller and endpoint application Test sequencer Keyword-driven test framework Python test framework Open-source test sequencer on .NET Open-source Python library for hardware tests Python library for embedded boards in a lab Continuous integration system for boards
Test is code in Python Sequence file Keywords Python XML test plan Python Python YAML job
Writing a structured test suite with AI agents The whole suite, from instructions written for AI agents Code modules; the sequence is built in the editor Keyword files, as text Test functions, as Python Plugins as code, the plan as XML Phases, as Python pytest tests, as Python Job files, as YAML
Running tests with AI agents across the lab On the machines the request names On the test station On its own machine On its own machine On its own machine On the test station On boards, from the command line Jobs submitted from the command line
Code first yes no no yes no yes yes no
Code once yes Limits and properties loaded from files Variables on the command line Options the test code adds Plan parameters and settings Configuration files Environment file A job file per device type
Code only yes no no yes no yes no no
GUI Built from the test code Sequence Editor, operator interfaces Separate editors, such as RIDE no Editor Web view of the test station no Web interface of the server
Remote machines in a run yes Windows computers with TestStand Through a remote library no no no Boards, driven from the test host Boards
Machine chosen per step yes Per called sequence Per keyword library no no no Per command, a board Per job
Test package upload yes Files on a shared path no no no no Files synced to the lab host Images and test definitions, by the job
Endpoint software update yes no no no no no no no
Live logs From every machine, in the GUI Execution view Console progress; the log after the run Terminal output Log of the run Web view Terminal output Job log in the web interface
Results tree At every level, from the session down to the step, with the configuration beside the results and a filter by status Report, per step Log file, down to the keyword Per test Per step, through result listeners Per phase and measurement Per test Per test case
Results export XLSX, session files, text for AI agents HTML, XML, ATML, database HTML, XML, xUnit JUnit XML; more through plugins CSV and databases, through result listeners JSON, output callbacks As pytest CSV, YAML, through the API
Machines with roles yes no no no no no Named targets in the environment file Roles in a multi-device job
Fan-out: one run per machine yes Per unit under test, parallel and batch models no no no no no A job per device
Machines shared by the team yes no no no no no Places, through the coordinator Devices, through the scheduler
Machine pools yes no no no no no Reservations by tags By device type and tags
Case matrix From dimensions: randomized, fetched at run time, with exclusion rules, filtered in the GUI Loops in the sequence Test templates Parametrized tests Sweep steps no As pytest no
Reboot inside a test yes no no no no no Power cycles the board Boots the board in the job
Run in a CI pipeline A snapshot of the run, exported from the GUI Through the command line or the API The same command The same command The same command The test script As pytest It is the CI system for boards
Past runs loaded into the GUI Runs from the command line, CI or another machine, with their configuration, to change and rerun Saved reports, to read Log and report files, to read in a browser no Results viewer, to read from a database Past tests of the station, to read in the web view no Every job, in the web interface; a job can be resubmitted
OS Windows, Linux, macOS Windows Windows, Linux, macOS Windows, Linux, macOS Windows, Linux, macOS Windows, Linux, macOS Linux Linux

Beside the tools around a run

Two kinds of tools do a different job, and Orcraft works together with them:

  • Test management tools such as TestRail, Xray and Zephyr: they track cases, plans and results; Orcraft runs the tests.
  • CI servers such as Jenkins, GitHub Actions and GitLab CI: they trigger the run; Orcraft carries it out on the lab's machines.

All other product names are trademarks of their respective owners and are used for identification and comparison purposes only. Orcraft is not affiliated with, sponsored by or endorsed by any of them. The descriptions of other products on this page compare features and functionality to help you decide whether Orcraft fits your needs; they are not intended to disparage any product or its owner. They are summaries written by us, reflect our understanding at the time of writing, and may be inaccurate or out of date. For an accurate description of any product, visit its own site. If you believe a description is wrong, tell us and we will correct it.

Type to search the docs.

    ↑↓ navigate ↵ open Esc close