results
Numbers from
a real AI PC.
Everything here was measured on an Intel LunarLake AI PC with an Arc 140V GPU and an NPU, running Windows 11, and is written down in the OVAT repository with how it was taken. Where a result is thin, it says so. A measurement that lies is worse than one that is missing.
Intel Core Ultra (LunarLake), Arc 140V GPU, Intel AI Boost NPU, Windows 11
OVMS 2026.2.1 vs 2026.4.1
One server upgrade, fourteen more right answers
Right answers out of 60
15 questions that need a tool (RAG, audio, vision, document Q&A and NPU) through all four engines. Qwen3.5-4B on the GPU and Qwen3-8B-int4-cw on the NPU.
Is it chance? McNemar's test, which compares paired right and wrong answers on the same questions, gives p = 0.0001. That is why OVAT pins OVMS 2026.4.1 in core/ovms_installer.py, and why ovat setup and doctor read the installed version.
The 59/60 run came later, on main, after round three fixed the vision tool. It is a separate run and was not part of the paired test.
Memory Read from the server process
A quarter of the memory, for the same work
GPU server peak, same 15 x 4 run
Peak memory of the OVMS process serving the GPU model.
Where a model's memory lives
Qwen3.5-4B-int4-ov served by OVMS. The weights sit in the server, not in OVAT.
The load peak sits 2.2 GB above steady state, and the peak is what decides whether a model fits. That is why 8 GB machines should use the 0.8B model. On the local ovat chat path the model runs inside OVAT, so there the trace figure is the model.
Determinism temperature 0
Same question, same answer, finally
Distinct answers to one repeated request
Each dot is one run. Same colour means the same answer text.
2026.2.1: up to 5 different answers in 5 runs at temperature 0, the worst case shown. A fixed seed: 42 did not help: 0 of 20 attempts matched across two identical arms.
2026.4.1: every engine returned one identical answer across 10 runs, on two servers. So on 2026.4.1, repeating one question gives one sample, not ten. The unit of measurement is a different question, which is why the accuracy test uses 15 of them.
Tuning measured, then left off
Extra settings did not beat the plain config
Right answers out of 60, OVMS 2026.4.1
The same 15 questions through all four engines, one setting changed at a time.
None of the three got a single answer right that the plain config got wrong (p >= 0.5), so all three stay off by default.
About auto: on OVMS 2026.2.1 it selected no parser at all and printed the raw tool call as the answer. On 2026.4.1 it detects qwen3coder. OVAT still names the parser, so an older server someone installed by hand also works.
KV cache static size, three runs per arm
A full cache costs about ten times the latency
Seconds for one LangChain tool-calling test
The KV cache is the model's working memory for a conversation. These runs fixed its size with --cache_size.
What it shows: a full cache reproducibly costs about 10x the time, which matches the preemption and recompute that OVMS documents.
What it does not show: that a full cache breaks tool calls. The failure appeared once at 100%, but the same reading also passed, so at three runs per arm that stays unproven. And unless the size is fixed, OVMS grows the cache as it goes, so "100% full" on its own means little.
NPU OVMS 2026.2.1, LunarLake
The NPU can run a tool-calling agent
OpenVINO/Qwen3-8B-int4-cw-ov compiled for the NPU and became available.
examples/document-qa-npu.yml over 2 turns, in 63.8 s, with no undecoded call.
A standard -int4-ov export fails to compile on the NPU before generating a token.
With --max_prompt_len 2000, every reply stopped at exactly this total.
Prompt plus reply always ended at 2129 tokens
The NPU pipeline is compiled to fixed shapes, so the total length has a ceiling however the split falls. The reply was cut mid-sentence and labelled finish_reason: "unknown", not "length".
A tool call still being written when the cap lands is cut in half, which looks exactly like a parser failure. For an agent, whose prompt grows every round, the prompt length setting matters most; OVAT raises the NPU default from 1024 to 4096.
Not verified: that group quantisation is the cause of the stock export's failure. The compiler's own message was about duplicated names. What is verified is that the channel-wise export compiles and the stock one does not.
Local chat Qwen3.5-0.8B
Telling a small model it has no tools
Right answers out of 3
The local path retrieves first and has no tools, but it used to pass on the workflow's prompt, which said "always call search_docs". The small model started each answer with the literal text "search_docs". One added note fixed it.
The note is _NO_TOOLS_NOTE in agent/rag_chat.py. Three questions is a small sample; it is reported as measured.
Benchmarks why each engine gets its own process
The lightest engine looked like the heaviest
Peak memory of the native engine, MB
Measured against live OVMS. Memory is a whole-process number, so engines sharing a process inherit each other's allocations.
Reversing the order reversed the conclusion. ovat bench now starts a fresh process per engine, and a failing engine is a row in the table, not a crash.
Four engines, one question, one server
An ovat bench run of the RAG example (the screenshot is in the guide): Qwen3.5-4B on the Arc 140V GPU, each engine in its own process. One run per engine, so read it as a snapshot, not a ranking.
Build time was 0.734 s for native, 15.185 s for react, 1.372 s for llamaindex and 1.132 s for openai-agents. All four answered ok, and all four recorded prompt and completion tokens from OVMS's usage field: 2002/485, 1410/387, 1456/224 and 1418/218, with 2, 1, 1 and 1 tool calls. An earlier run of the same example answered in 14.6, 164.9, 17.6 and 10.3 s, so a single run says little about which engine is faster.
Quality every push
800+ tests, chosen platforms
The suite needs no server and no model; tests that do are marked and skip themselves. Every fix ships with a test that fails when the fix is backed out. The CI matrix is picked to open as many differences between machines at once as possible, because every bug the project shipped lived in a gap between two machines.
| Runner | Python | Why it is there |
|---|---|---|
| ubuntu-22.04 | 3.10 | The oldest of everything supported, including SQLite 3.37, where RAG was once silently empty. |
| ubuntu-24.04 | 3.12 | A current Linux. |
| windows-latest | 3.12 | The only platform with .exe and backslash paths. The suite once passed only here. |
| macos-latest | 3.13 | The development platform. |
Test functions, week by week
Counted on the main branch every seven days.
Lines in the package, week by week
Python lines under ovat/, same snapshots.
Test functions per test file
The 33 files in tests/, grouped in tens. Counts are def test_ functions; pytest runs more cases than this, because some tests are parametrized.
The flat stretch from late August to early October is the project between review rounds. Counted with git rev-list and git grep on the OVAT repository on 9 October 2026.
Two more jobs check what the suite itself cannot: that a base install imports no framework and no UI library, and that the built wheel installs and runs. Runners are not root on purpose: the OVMS installer once passed in every root environment and failed for the first real user.