ovat navigate

guide

From pip install
to a real answer.

Every command in the order you will need it. Each step says what it does and what you should see. If anything goes wrong, ovat doctor is the fastest diagnosis: every yellow row says what to do.

Install

You need Python 3.10 to 3.14. The full agent needs Windows 11, Ubuntu 22.04 or 24.04, or RHEL 9 on x86-64, because that is where OVMS runs. macOS works for everything except serving. Plan for 8 GB of RAM at least, 16 GB to be comfortable, and about 8 GB of disk (15 GB with RAG).

terminal
$ pip install ovat

The base install already has the native engine, all built-in tools, MCP, RAG and telemetry. Extras add only what you ask for:

Optional extras
ExtraGives you
langchain llamaindex openai-agentsThe matching agent.type
tuiThe full-screen terminal UI (ovat with no arguments)
convertoptimum-cli, to convert Hugging Face models to OpenVINO IR (needed for the RAG embedder)
devEvery framework above, plus pytest
everything at once
$ pip install "ovat[langchain,llamaindex,openai-agents,tui]"

Linux needs two system packages first. A minimal image has no Python, and OVMS needs libxml2. On Ubuntu: sudo apt install -y python3 python3-venv python3-pip libxml2 curl. On RHEL 9: sudo dnf install -y python3 python3-pip libxml2 curl.

Using a GPU or NPU? Update the driver first. An old driver usually shows up as the device being missing from ovat doctor, not as an error. On Windows also install the Visual C++ Redistributable, which OVMS needs to start at all.

Your first run

  1. Install the model server, once

    terminal
    $ ovat setup

    Picks the right OVMS archive for your OS, checks its SHA-256, and unpacks it into ~/.ovat/ovms. No PATH edit, no environment variable. Running it again does nothing. On macOS it explains that there is no build and what to use instead.

  2. Write a starter config

    terminal
    $ ovat init workflow.yml

    Writes a workflow.yml tuned to this machine, with the device it detected.

  3. Check the machine and the config

    terminal
    $ ovat doctor workflow.yml

    A table of checks: Python, dependencies, OpenVINO devices, device routing, OVMS, and your config. Green is fine, yellow is a heads-up with the fix written next to it, red needs fixing.

    ovat doctor on the AI PC for the RAG example: twelve checks, all ok, including CPU, GPU and NPU detected, all three framework engines installed, OVMS found where ovat setup put it, and OVMS reachable.
    On the Intel AI PC, for examples/rag/workflow.yml: 12 ok, all clear.

  4. Prove the wiring, with no server

    terminal
    $ ovat run workflow.yml --dry-run
Built agent  model=Qwen3.5-4B-int4-ov  tools=['search_docs', 'transcribe']  max_iterations=10
dry-run: not calling the model.

    Builds the whole agent and stops before calling the model. Works on any machine, including a Mac.

  5. Start OVMS

    terminal
    $ ovat serve workflow.yml

    The first time, this downloads the model: 3.5 GB for the default. It shows elapsed time and only gives up after five minutes with no progress at all, so a slow connection is fine. When OVMS is ready, the prompt comes back and the server keeps running in the background. Its process ID is saved in ovms.pid and its log in ovms.log.

  6. Ask something

    terminal
    $ ovat run workflow.yml -i "summarise my notes"

    The first line says which engine ran. Then comes the answer, and, when a document search ran, a sources: line that OVAT prints from the tool results itself. Add --react, --llamaindex or --openai-sdk to try another engine without editing the file.

  7. Give it your documents

    terminal
    # the embedder is the one model you convert yourself
$ pip install "ovat[convert]"
$ optimum-cli export openvino --model BAAI/bge-small-en-v1.5 \
    --task feature-extraction models/bge-small-en-v1.5

$ ovat index ./my-notes workflow.yml

    Reads every .txt and .md file in the folder, slices it into chunks, embeds them and stores them in the file named by rag.retriever.db_path. Run it again after editing documents: run and chat warn when an indexed file has changed or gone.

  8. Shut it down

    terminal
    $ ovat serve workflow.yml --stop

    Stops the OVMS that serve started, after checking the saved process really is OVMS.

On a Mac: chat locally, no server

OVMS has no macOS build. openvino_genai does run natively on Apple Silicon, so ovat chat gives you real retrieval-augmented answers on the CPU: it embeds your question, pulls the closest chunks from your index, and has a local model answer from them. It does not call tools; it always retrieves first and then answers once.

macOS, from a clone of the OVAT repo
$ git clone https://github.com/Lagmator22/ovat.git && cd ovat
$ hf download OpenVINO/Qwen3.5-0.8B-int4-ov --local-dir models/Qwen3.5-0.8B-int4-ov
$ pip install "ovat[convert]"
$ optimum-cli export openvino --model BAAI/bge-small-en-v1.5 \
    --task feature-extraction models/bge-small-en-v1.5
$ ovat index ./examples/rag/docs examples/rag/workflow.yml
$ ovat chat examples/rag/workflow.yml -i "What is OVAT's memory budget?"

ovat chat finds a local model by itself (in model_search_paths, OVAT_MODELS, ./models and ~/models), or take one with --model-path. It streams the answer; Ctrl-C stops it. There is no answer length cap unless you set --max-tokens.

Compare the four engines

terminal
$ ovat bench workflow.yml -i "What can you do?" --out report.json
$ ovat bench workflow.yml -i "..." --repeat 5     # success rate and medians
ovat bench on the RAG example on the Intel AI PC, Qwen3.5-4B-int4-ov: native, react, llamaindex and openai-agents all ok. Answer times 18.6, 14.7, 9.7 and 8.0 seconds; peak memory 454.1 to 664.4 MB; tokens in and out 2002/485, 1410/387, 1456/224 and 1418/218; tool calls 2, 1, 1 and 1.
The RAG example on the AI PC, Qwen3.5-4B on the GPU. All four engines answer ok, and all four record prompt and completion tokens from OVMS's usage field, plus tool counts. A dash would mean unknown, never zero.

Each engine runs in its own fresh process, so one engine's memory does not leak into the next one's number. A failing engine becomes a row with the reason written under the table, not a crash.

Traces and telemetry

terminal
$ ovat run workflow.yml -i "..." --trace trace.json        # one run: turns, tokens, tools, latency
$ ovat run workflow.yml -i "..." --telemetry live.jsonl  # sampled while it runs
$ ovat telemetry                                            # the machine, live, no run

The trace records every turn: the tool calls with their sources, token counts, latency, and the failure flags (truncated, undecoded_tool_call, empty_answer). A failed run still writes its trace and exits non-zero. Unknown values stay null, never 0.

ovat telemetry and the terminal UI's telemetry page read the same sources. On the AI PC every source is live, including Intel's own counters for package and rail power:

The telemetry page in the OVAT terminal UI on the AI PC: big readouts for NPU, KV cache, system RAM, OVAT memory and OVAT CPU, then a table of live numbers with now, min, max and mean, including intel rows for package power and NPU bandwidth. Every source says live.
The live page: system, process, NPU, OVMS and Intel sources all live.

One caveat on memory: --trace measures the OVAT process. With OVMS serving, the model lives in the server process. For Qwen3.5-4B that was measured at 4.3 GB steady and 6.5 GB peak, while the trace said about 0.45 GB. On the local ovat chat path the trace is the model.

The terminal UI

Install the tui extra, then run ovat with no arguments. Anything you type runs as a real command in your environment; typing / opens a menu of OVAT shortcuts. Chat runs inside the app: the model loads once, answers stream, and conversations are saved in .ovat/sessions/.

The OVAT terminal UI launcher: the OVAT, powered by OpenVINO OVMS masthead, a list of recent updates, the Intel mark, and an output pane that says any command runs in the venv and slash opens OVAT shortcuts.
The launcher. Type / for shortcuts, or any command to run it in your environment.
The chat screen switched to OVMS with tools. The question asks about an audio file, an image and the documents; the answer gives the transcript, describes the Intel logo, and reports what document search returned. A footer lists the tools used: transcribe, describe_image and search_docs.
One question, three tools: transcribe, describe_image and search_docs. This workflow has no rag: section, so document search answers in its marked stub mode, and the answer says so.
Launcher commands
LauncherDoes
/chatChat with your docs; optional config and model path
/doctorThe doctor checks, live; press r to re-run
/telemetryLive numbers for the machine and the run
/run /index /initFill in the matching ovat command
/serve /modelsStart OVMS, list its models (AI PC)
/clear /exitClear the output, leave
Chat screen commands
In chatDoes
/engineSwap between the local model and OVMS with tools
/thinkingShow or hide the model's reasoning
/tokens NAnswer length cap; 0 means none
/copyCopy the last answer, or me, or all
/save /loadSave or pick a conversation
Esc Ctrl+CStop the answer that is streaming

The CLI never needs the UI. textual and pyfiglet live only in the tui extra, and a test fails if a base install imports them.

Writing workflow.yml

Four sections: model (required), tools, agent and rag. Unknown keys are errors, so a typo fails immediately. Select a line or a note to see what it does. Every field, with its type and default, is in the generated reference.

workflow.yml, a complete example
model:  name: Qwen3.5-4B-int4-ov  source_model: OpenVINO/Qwen3.5-4B-int4-ov  device: GPU  tool_parser: qwen3coder tools:  - name: search_docs    type: builtin  - name: transcribe    type: builtin    model: models/whisper-base-int8-ov    device: CPU agent:  type: native  max_iterations: 10  system_prompt: >-    Always call search_docs before answering a question about    the user's files, and cite the source path you used. rag:  embeddings:    provider: genai    model: models/bge-small-en-v1.5    dim: 384  retriever:    provider: sqlite-vec    db_path: ovat_index.db

Name your tool parser. The wrong one fails silently: the agent answers fluently and never calls a tool. If you leave the field out, OVAT derives it from the model name, and falls back to hermes3 for families it does not know.

Adding an MCP tool

Any MCP server that speaks over stdio becomes a tool with a few lines of YAML. OVAT launches it, asks which tools it offers, and imports every one of them. Here is a tool of your own, written with FastMCP, the same library OVAT's built-ins use.

my_tool.py
from fastmcp import FastMCP

mcp = FastMCP("word_count")


@mcp.tool
def word_count(path: str) -> int:
    """Count the words in a text file.
    Use when the user asks how long a file is."""
    with open(path, encoding="utf-8") as f:
        return len(f.read().split())


if __name__ == "__main__":
    mcp.run()
workflow.yml
tools:
  - name: word_count
    type: mcp_stdio
    command: ["python", "my_tool.py"]

  # someone else's server, from npm
  - name: filesystem
    type: mcp_stdio
    command: ["npx", "-y",
      "@modelcontextprotocol/server-filesystem", "/docs"]

The docstring is for the model. It reads the description to decide when to call your tool, so say plainly when to use it.

Environment. An MCP server starts with only a few variables (HOME, PATH and a handful more). Pass anything else with env: under the tool; ${VAR} is expanded from your shell.

It works the other way too. OVAT's own tools are MCP servers: python -m ovat.tools.transcribe gives any MCP-aware agent OpenVINO Whisper, and python -m ovat.tools.search_docs --config workflow.yml serves your index.

Picking a model

These are ready-made OpenVINO exports on Hugging Face. ovat serve downloads the one in source_model; on a Mac fetch it with hf download.

Models
ModelDownloadMemoryUse it when
Qwen3.5-4B-int4-ov3.5 GB4.3 GB steady, 6.5 GB peak (measured)The default. Text, vision and tools in one model.
Qwen3.5-0.8B-int4-ov0.9 GBabout 2 GB (estimate)An 8 GB machine, or a fast first run.
Qwen3-8B-int4-ov4.9 GBabout 6 to 7 GB (estimate)Strongest text answers; no vision.
Qwen3-8B-int4-cw-ovThe NPU. Channel-wise INT4, the export the NPU compiles.
whisper-base-int8-ov0.08 GBsmallThe transcribe tool.

Only the Qwen3.5-4B memory figure was measured, on an Intel AI PC, from the OVMS process. The others are estimates from parameter count and precision. The load spike decides whether a model fits, so 8 GB machines should prefer the 0.8B model.

Troubleshooting

Symptoms and likely causes
SymptomLikely cause
The agent answers fluently but never calls a toolWrong tool_parser. Qwen3.5 needs qwen3coder, Qwen3 needs hermes3. OVAT warns when raw tool-call markup reaches the answer.
OVMS exited without becoming ready, empty logThe python_off build, or a missing VC++ Redistributable on Windows.
ovat doctor finds no OVMSSet OVAT_OVMS to the unpacked folder, or run ovat setup.
No GPU or NPU listed in doctorThe driver is out of date.
search_docs returns [stub]No rag: section, or no --config on the MCP command.
serve looks stuckThe first run downloads the model. Watch ovms.log.
A request times out on a CPU-only machineRaise model.request_timeout; an agent turn is several model calls. OVAT says this when it happens.
NextSee what was measured

Accuracy, memory, determinism and the NPU, on a real AI PC.

 OrHow it works inside

Each part and what it does, animated.