Structured index: how an AI agent answers "how many" about your documents
A structured index is a small table an agent builds from the fields your documents already carry — title, price, level, link, image — alongside the vector search it uses for meaning. Vector search finds passages that read like the question; it cannot count, filter by number, or group. The structured index answers those, and lets the agent quote a link or an identifier exactly instead of reconstructing it from a fragment.
Ask "how many do you have?" and the agent answers from the four passages it happened to retrieve. Ask for a link and it may hand you a real one belonging to a different item — it opens, the image renders, nothing errors. Here is how that happens, and what stops it:
Each document owns a colour. On the left — retrieval as it works today — the answer's title comes from one document and its link from the other, and the colours give the mismatch away instantly. On the right, each retrieved fragment arrives carrying its own document's facts, and the link is not a link at all: it is a placeholder the model copies without ever seeing the address. The real value is substituted on the way out, so there is nothing to mix up.
Retrieval finds passages; it does not see the collection
Vector search works by similarity: your question becomes a direction in a space, and the passages closest to it come back. That is exactly right for "what does this say about X" and structurally useless for "how many", "which ones are under 200 words", or "how many per category" — those are questions about the whole collection, and retrieval only ever looks at the nearest few.
No amount of tuning fixes this. There is no threshold that makes a similarity search return a total, because a total is not a passage.
The second, quieter failure: a chunk is not a document
Long documents get split into chunks so retrieval can find the right paragraph. But your title, your product code, your link and your image live in the header — which means they live in chunk 1, and the passage that actually answered the question may be chunk 7.
So the model ends up holding four or five fragments from three or four different documents, and it has to work out which link belongs to which title. It guesses. And when it guesses wrong the answer does not look wrong: the link is real, it opens, the image renders — it just belongs to a different item. Nothing errors. Nobody notices.
On a 4,500-document catalogue, 11 of 20 test answers paired a title with another document's link. With the placeholder mechanism above, the same 20 came back 20 of 20 correct.
What a structured index is
It is a small table, one row per document, built from the fields your documents already contain. Nothing is invented and nothing is rewritten — the values are copied out character for character.
With it in place, the agent has two ways to answer instead of one:
| Question | Answered by |
|---|---|
| "Do you have anything about dinosaurs?" | vector search |
| "How many do you have?" | the table |
| "Something about dinosaurs, suitable for a five-year-old" | the table filters, vector search ranks |
| "What's the link and the cover image?" | the table, quoted exactly |
You do not define the fields, but you do get the last word
The platform reads a few of your documents, proposes the fields, and then verifies each one against your whole collection before keeping it. A field that only turns up in a fifth of your documents is dropped and reported, because filtering on it would silently omit the rest.
Two kinds of input are yours to give:
Which fields must be exact. Which link is the link to send a customer, which of three codes is the one they will quote back at you — your data does not state this, and it cannot be inferred. Name them up front and they are extracted verbatim and quoted verbatim.
Corrections, in plain language. "Also track the author so people can find other books by them." "I want to filter by illustrator." "Drop the cover link." Changes are proposed as a patch to the existing table, never a rewrite, so a request about one field cannot disturb another. Ask for something your documents do not contain — a publication year that was never in the export — and it tells you so rather than adding an empty column.
What it also tells you about your own data
Because every field is measured across the whole collection before it is accepted, the report regularly surfaces things nobody knew. On one real catalogue of several thousand titles it found that a tenth of them had a page count of zero — not short books, but missing values recorded as zero, which would have dragged down every average and produced a large tie for "shortest".
That check has to happen somewhere. Today it usually happens when a customer notices a wrong answer.
When it does not apply
If your documents are prose — articles, transcripts, correspondence — there is nothing to tabulate, and the platform declines to build an index rather than producing one made of values that are all different. A table like that cannot be counted or grouped in any meaningful way, and having it would only invite questions it answers badly.
Vector search remains the right and only tool for that material.
Common questions
- Why can't my agent tell customers how many products I have?
- Because vector search returns a handful of passages that read like the question, and a handful of passages cannot be counted. It is not a tuning problem — there is no setting that makes retrieval return a total. A structured index answers counting, range and grouping questions by querying a small table built from the fields your documents already carry, while vector search keeps handling questions about meaning.
- Will the agent invent a link or a product code?
- Not when the field is marked as exact. Those values are never shown to the model at all: it receives a placeholder, and the platform substitutes the stored value on the way out. The model cannot mistype a URL it never saw. This matters more than it sounds — the common failure is not an invented link but a real one belonging to a different item, which opens fine and looks correct.
- What kind of documents can this be built from?
- Documents that share a machine-readable header: a metadata table, YAML front matter, or Field: value lines. Product exports, catalogues, drug leaflets, policy schedules and course listings usually qualify. Plain prose does not, and the platform says so instead of building a useless table — a table of values that are all different cannot be counted or grouped meaningfully.
- Do I have to define the fields myself?
- No. The platform reads a few of your documents and proposes the fields, then verifies every one of them against your whole collection before keeping it. You can add, rename or drop fields afterwards in plain language, and you can name up front which fields must be extracted exactly — the title, the link, the image, the code — because which link is the one to send a customer is a business fact your data does not state.
- Does adding this re-process my knowledge base?
- No embeddings are recomputed. The fields are read straight out of the text your documents already contain, so building or changing the index costs no re-indexing and does not disturb the vector search that is already working.
- How does it stay current when I add or remove documents?
- New documents are read as they are imported, using the rules already established, so nothing waits on a model. The index itself is rebuilt on demand before the next question that needs it — adding a thousand documents does not trigger a thousand rebuilds.
Most of what an agent needs to know is already written down — on your website, and in the PDFs your team already sends customers. Importing from a URL covers the public half; uploading covers the rest.
A storyline is a directed graph an agent follows with each end user — every node is a step (its own task, knowledge and tools), exits carry conditions, and each person's progress, profile and notes are saved and resumed across sessions and channels. It turns an agent from an assistant that does point-tasks into one that can deliver a multi-step service itself.
The Dynamic Planner watches a conversation and, when a user is pursuing a genuinely multi-step task inside the agent's domain and is visibly unsure how to proceed, offers to expand that task into a temporary storyline — a checklist-driven, step-by-step plan the agent then executes one focused step at a time, with progress the user can see. Planning happens once, under validation; execution is deterministic.