Skip to content
Chatvia Docs
On this page

API reference

Evals

Get in touch: Ahmad Mayahi, creator of Chatvia, Spectra and Shakomako

These endpoints start an agent's evals and read the results. An eval is a test question with a check on the answer, for example "the answer to 'What is your refund window?' must contain '30 days'". You write the test cases on the agent's Evals page in the dashboard. A run asks the agent every active case and scores each answer.

For example, your deploy pipeline can start a run after someone changed the agent's instructions, wait for it to finish, and stop the release when the pass rate is below 90 percent.

Read the overview first for the base URL, keys, errors and paging.

The run object

json
{
  "id": "0199f430-3b4c-7d5e-8f6a-7b8c9d0e1f2a",
  "status": "completed",
  "total_cases": 20,
  "passed_count": 19,
  "failed_count": 1,
  "avg_score": 0.93,
  "pass_rate": 95,
  "total_tokens": 48210,
  "model_snapshot": "GPT-4.1 mini",
  "started_at": "2026-10-25T13:00:00+00:00",
  "finished_at": "2026-10-25T13:01:42+00:00",
  "created_at": "2026-10-25T13:00:00+00:00",
  "results": null
}
Field Type Meaning
id string The run's id.
status string running while cases are scored, then completed or failed.
total_cases integer How many cases the run scores.
passed_count, failed_count integer Cases scored so far as passed and failed.
avg_score number or null The average score of the scored cases, from 0 to 1, when the checks give scores.
pass_rate integer or null The share of scored cases that passed, in whole percent.
total_tokens integer or null The tokens all answers used together.
model_snapshot string or null The AI model the agent used when the run started.
started_at, finished_at string or null When the run started and finished. finished_at is null while it runs.
created_at string When the run was created.
results array or null The scored cases. Filled on Get a run only. null in the list.

List runs

Returns the agent's eval runs, newest first.

text
GET /api/v1/agents/{agent}/evals/runs

Who can call it: the owner, or a teammate with View evals.

bash
curl https://chat.example.com/api/v1/agents/a3f9c2e17b4d08/evals/runs \
  -H "Authorization: Bearer $CHATVIA_API_KEY"

The answer is { "data": [run, …], "next_cursor": … }. See Pagination.

Start a run

Starts a run over the agent's active cases. The cases are scored in the background.

text
POST /api/v1/agents/{agent}/evals/runs

Who can call it: the owner, or a teammate with Manage evals. This endpoint has its own rate limit of 6 a minute. It takes no body.

bash
curl -X POST https://chat.example.com/api/v1/agents/a3f9c2e17b4d08/evals/runs \
  -H "Authorization: Bearer $CHATVIA_API_KEY"

The answer is 201 Created with { "data": run }. The status is running, or completed right away when the agent has no active cases.

Good to know:

  • Every scored case counts as one message toward your workspace's message limit. When the workspace has reached its limit, the run doesn't start and the answer is a 422.
  • A run scores at most 100 cases. The admin of your Chatvia server can change this number with EVALS_MAX_CASES in the environment file.

Get a run

Returns one run with every case scored so far, oldest first. Call it every few seconds until finished_at is set.

text
GET /api/v1/agents/{agent}/evals/runs/{run}

Who can call it: the owner, or a teammate with View evals.

bash
curl https://chat.example.com/api/v1/agents/a3f9c2e17b4d08/evals/runs/0199f430-3b4c-7d5e-8f6a-7b8c9d0e1f2a \
  -H "Authorization: Bearer $CHATVIA_API_KEY"

The answer is { "data": run } with results filled:

json
{
  "id": "0199f431-6c7d-7e8f-9a0b-1c2d3e4f5a6b",
  "case_name": "Refund window",
  "scorer_type": "contains",
  "pass": true,
  "score": null,
  "actual_output": "You can ask for a refund within 30 days of delivery.",
  "actual_output_truncated": false,
  "sources_used": ["returns-policy.pdf"],
  "tools_called": [],
  "judge_rationale": null,
  "latency_ms": 1830,
  "tokens": 2115,
  "created_at": "2026-10-25T13:00:21+00:00"
}
Field Meaning
id The result's id.
case_name The case's name when it was scored.
scorer_type How the answer was checked: contains, regex, exact, json_schema, llm_judge, tool_call or citations.
pass Whether the case passed.
score A score from 0 to 1, when the check gives one.
actual_output The agent's answer. Long answers are cut for storage.
actual_output_truncated Whether the answer was cut.
sources_used The sources the agent found for the answer.
tools_called The tools the agent called.
judge_rationale Why an AI judge passed or failed the answer, for llm_judge cases.
latency_ms How long the answer took.
tokens The tokens the answer used.
created_at When the case was scored.

A simple check in a pipeline: start a run, keep its id, call this endpoint until finished_at is not null, then compare pass_rate with your threshold.

Search guides, settings and commands.