From source to knowledge#

When you add a source to an agent and train it, Chatvia turns it into small, searchable parts called chunks. This page explains each step. All of it runs in the background, so you can keep working while it happens.

Step 1: Read the text#

Each kind of source is read in its own way.

Some checks happen before training starts:

• A PDF locked with a password is refused. Remove the password and upload it again.
• The same file or text can only be added once per workspace.

The admin can turn file types on or off in Settings → Knowledge → File types.

Step 2: Split the text into chunks#

An AI model can only read a limited amount of text at once, and a search works best on small, focused parts. So Chatvia splits every source into chunks.

How it splits depends on the kind of content:

The default chunk size is 400 tokens (about 300 words), with no overlap between chunks. A token is the unit AI models read text in. One token is about three quarters of an English word.

The admin can change the size, the overlap and the way each kind of content is split, on the admin panel's Settings → Splitting page. Its Try splitting card shows the result on your own text before you save. A new setting applies to sources the next time they are trained.

Step 3: AI enhancement (optional)#

This step makes chunks a little easier to find, for extra AI calls. It is off by default, and you can turn it on in Settings → Knowledge → AI enhancement. It has two switches: one for files and web pages, and one for text sources.

For each chunk, an AI model writes, in the chunk's own language:

• A short summary (2 to 5 sentences), which tells the search what the chunk is about even when the chunk itself is short or vague.
• 5 to 7 questions the chunk answers. A chatter's question often looks more like one of these questions than like the chunk text, so they are searched too.

For documents, the model also sees the page before and after, and two more things happen:

• Pictures are described in words, when Describe images is on in Settings → AI models → Documents and a model that reads images is set up. Pictures with no useful content, like logos and decoration, are dropped. The rest are kept, so the agent can show them in an answer.
• Tables are turned into plain sentences, which search better than a grid.

The model for this step is set in Settings → AI models → Models for app tasks → AI enhancement. Tables have their own model there (Tables), and pictures use the Model for image descriptions on the Documents card. With enhancement off, training makes no calls to a chat model, and it is faster. Search still works, but it finds the right chunk less often.

If enhancing a page fails three times, the page is stored without enhancement instead of being lost.

How much it helps#

These numbers come from our search evaluation: an automatic test that asks the search a fixed set of questions and counts how often the right text comes back.

How we measured it:

• The knowledge. Twelve documents of one made-up outdoor shop: help pages, a product catalogue, a footwear catalogue, terms of sale, guided trips, hiking routes, a guide for businesses, a video transcript and three store pages in Arabic. They split into 148 chunks.
• The questions. 244 questions we wrote by hand, in English and Arabic. For each one we wrote down the text the right chunk must contain. Some questions are about things the knowledge does not cover, so the test also sees if the search correctly finds nothing.
• The runs. The same questions ran four times: with AI enhancement on and off, each with and without a reranker. With AI enhancement on, the sources were learned again, so every chunk got its summary and questions.
• What counts. The test checks if the agent gets the chunk that holds the answer. It does not judge the written answer.

The settings, the same in every run:

Each kind of question runs twice: once with a reranker and once without. To see what AI enhancement changes, compare the last two columns of a row.

What this means:

• AI enhancement finds the right chunk more often for questions with one answer. With a reranker, it answers 7 more questions out of 168. Without a reranker, it answers 2 more.
• It helps most when a question and the text use different words or languages. Without it, none of the English questions about the Arabic store pages found their chunk among the candidates; with it, they did, though most still ranked below the top 5.
• It does not change how often the agent correctly finds nothing. Without a reranker, the agent gets text for more than half of the questions the knowledge does not cover, because many of them sound close to it. A reranker blocks most of them.
• For questions that need several facts, it makes little difference: one question more or less, either way.

The gaps above are small: one question is about 0.6% of the 168.

Step 4: Embeddings#

Next, Chatvia sends each chunk to the embedding model. It returns an embedding: a long list of numbers that captures the meaning of the text. Texts with a similar meaning get similar numbers, even when they use different words. This is what makes search by meaning possible.

Chatvia makes one embedding for the chunk (with its summary and picture descriptions) and, when enhancement is on, one for its questions.

Pick the embedding service in Settings → AI models → Embeddings. Chatvia supports Qwen3 on Scaleway (the default), OpenAI, OpenRouter, Google Gemini, Cohere, Mistral, Jina and Voyage.

The size of the stored embeddings is chosen once, when you install Chatvia (EMBEDDING_DIMENSIONS, 1024 by default). It can't change later, so the embedding model you pick must be able to return that size.

Step 5: Index for search#

The chunks are stored in PostgreSQL with two kinds of search index:

• An index for search by meaning on the embeddings (pgvector HNSW). By default it holds a short form of each embedding (1 bit per dimension), which makes it about 6 times smaller. Search finds the closest matches in it, then checks them against the full embeddings. See Embedding storage.
• An index for search by keywords on the text and the questions.

The keyword index has two possible forms, depending on your database. See The two keyword search engines.

Step 6: Summary of the whole source (optional)#

When AI enhancement is on and Source summaries is on in Settings → AI models → Documents (it is by default), Chatvia also reads the whole source once more, with a model that reads PDF files. Pick that model in Model for source summaries, or leave it empty to let Chatvia choose one. It writes a summary, detects the language and the kind of document (a FAQ, a user manual, a contract…) and stores them with the source.

Progress and status#

While a source trains, the Sources tab shows its progress live. A source moves through these states (the exact words differ a little between the source lists):

If only some chunks fail, the source still finishes, and the chunks that were stored can be searched. You get a notification in the dashboard bell when a source finishes training or fails.

Training again#

• Text sources are trained again when you edit them.
• Web pages can be trained again by hand, or on a schedule (daily, weekly or monthly). A page that has not changed since the last time is skipped, so no work is done for it.
• Deleting a source removes its chunks at once, so agents stop using it right away.

Q&A pairs are special#

Q&A pairs are not chunks. Each question is stored on its own, with its embedding. Before every search, Chatvia checks the chatter's message against your saved questions:

1. The same question. The message matches a saved question, ignoring case, accents, punctuation and spacing.
2. The same meaning. The message is close enough in meaning to a saved question. This works with an embedding model that has a known similarity score, like Qwen3, or when the admin sets Saved answers: minimum similarity in Settings → Search.

On a match, the agent sends your saved answer word for word. No search runs and no chat model is called, so the answer is instant and always the same. This makes Q&A pairs the right tool for answers that must never change, like opening hours or legal wording.