Orcraft vs LAVA
Both run tests on the hardware of a lab from one place and keep the results. LAVA is built to deploy an operating system image onto a board, boot it and test it. Orcraft is built to run the steps of a Python test on the machines of the lab.
See the note on other products
Side by side
yes · no · words where a tool does it in its own way
| Orcraft | LAVA | |
|---|---|---|
| What it is | Test framework, controller and endpoint application | Continuous integration system for boards |
| Test is code in | Python | YAML job |
| Writing a structured test suite with AI agents | The whole suite, from instructions written for AI agents | Job files, as YAML |
| Running tests with AI agents across the lab | On the machines the request names | Jobs submitted from the command line |
| Code first | yes | no |
| Code once | yes | A job file per device type |
| Code only | yes | no |
| GUI | Built from the test code | Web interface of the server |
| Remote machines in a run | yes | Boards |
| Machine chosen per step | yes | Per job |
| Test package upload | yes | Images and test definitions, by the job |
| Endpoint software update | yes | no |
| Live logs | From every machine, in the GUI | Job log in the web interface |
| Results tree | At every level, from the session down to the step, with the configuration beside the results and a filter by status | Per test case |
| Results export | XLSX, session files, text for AI agents | CSV, YAML, through the API |
| Machines with roles | yes | Roles in a multi-device job |
| Fan-out: one run per machine | yes | A job per device |
| Machines shared by the team | yes | Devices, through the scheduler |
| Machine pools | yes | By device type and tags |
| Case matrix | From dimensions: randomized, fetched at run time, with exclusion rules, filtered in the GUI | no |
| Reboot inside a test | yes | Boots the board in the job |
| Run in a CI pipeline | A snapshot of the run, exported from the GUI | It is the CI system for boards |
| Past runs loaded into the GUI | Runs from the command line, CI or another machine, with their configuration, to change and rerun | Every job, in the web interface; a job can be resubmitted |
| OS | Windows, Linux, macOS | Linux |
Jobs and suites
LAVA is Linaro's open-source continuous integration system for boards. A job is a YAML file submitted to the LAVA server. It describes three things for one device type: how to deploy the image, how to boot the board and which tests to run on it. A dispatcher on a lab host carries the job out, and the server tracks the results. The system needs a server and dispatchers, installed and administered.
An Orcraft test is Python code that runs on any registered machine. The controller reads the suite and builds the GUI from it: the tree of tests, the fields that configure a run, the plan and the results. The Framework section describes each level of a suite.
Machines
LAVA's machines are boards: for each job a dispatcher deploys an image onto a board and boots it.
An Orcraft endpoint is a machine that is already running and can reach the hardware: a lab PC wired to a device under test, a server on the same network as the instruments, a machine with the vendor's tools installed. Every step names the machine it runs on, so one test can act on several machines in order, and a test can reboot its machine and continue when it comes back.
AI agents
A LAVA job is a YAML file, which an AI agent can write and submit. An Orcraft test is Python code in a defined structure: the AI agent writes the test logic itself, configures a run, runs it on the machines the request names and reads the results. Every part of the work can be done by a person or by an AI agent. The AI Agents section shows how.
All other product names are trademarks of their respective owners and are used for identification and comparison purposes only. Orcraft is not affiliated with, sponsored by or endorsed by any of them. The descriptions of other products on this page compare features and functionality to help you decide whether Orcraft fits your needs; they are not intended to disparage any product or its owner. They are summaries written by us, reflect our understanding at the time of writing, and may be inaccurate or out of date. For an accurate description of any product, visit its own site. If you believe a description is wrong, tell us and we will correct it.