AGENSPHERE/ GLOSSARYThe words we use to scope, build and argue about AI systems, each filed under the layer of the stack where it lives. Plain definitions first, then why the term matters and what we think about it.
Agents can handle work with no fixed path. They also multiply cost, latency and risk with every step they take on their own.
Use an agent only where the path really cannot be fixed in advance, and give it a budget, a stop rule and a record of every step.
When a customer, regulator or auditor asks why the system acted, you need an answer that does not depend on the model remembering.
Log inputs, retrieved sources, model versions, tool calls and approvals together, per request, from the start.
If a chunk cuts a table in half or drops its heading, the model receives half an answer. Chunking quality caps retrieval quality.
Chunk by the document’s structure, such as sections, clauses and tables, not by a fixed token count.
Everything the model knows for a request has to fit here. Bigger windows cost more and do not guarantee the model uses what is in them.
Decide what goes into the context on purpose. Filling the window is not a memory strategy.
Token prices say little on their own. Cost per task includes retries, retrieval, long contexts and agent loops, which is what actually shows up on the invoice.
Measure cost per task before you optimise anything. Most savings come from routing and caching, not from haggling over model prices.
Answers are only as fresh and as correct as the last pipeline run. Stale or duplicated data turns into confident, outdated answers.
Version the pipeline like code and track freshness per source, so you can say how old any answer’s sources are.
Without a stated policy, the line between suggesting and acting is decided by the model on each request.
Write the policy down, enforce it in code and log every decision against it.
Agents run for minutes or days, call slow tools and wait on people. Without durable execution, one crash halfway through means repeating work, or repeating actions.
Record every step and replay from the record. This is the idea behind KEEL, our durable execution engine for agents.
Embeddings make search by meaning possible. They are also the reason a system finds a passage that sounds right but is not the one the user needed.
Pick an embedding model by testing it on your own documents and questions. Public leaderboards rarely look like your data.
Without evals, every prompt edit or model upgrade is a guess, and you hear about regressions from customers.
Write the eval set before the prompt. It is the specification of what good looks like.
Providers have outages, rate limits and slow days. A product that only works when everything works is not ready for production.
Design a useful fallback for every model call, even if it is a plain message and a human handoff.
Corrections, edits and rejections are the best evaluation data you will get. Without a loop, the system stays as good as it was on launch day.
Capture feedback where the work happens, and turn it into new eval cases every week.
Fine-tuning changes style, format and narrow skills well. It is a poor way to teach a model facts that change.
Reach for retrieval and better prompts first. Fine-tune when you have a stable task, an eval set and enough clean examples.
Large language models (LLMs) are the reasoning engine of most AI systems today. They are powerful, general and stateless, and each provider’s model behaves a little differently.
Treat the model as a replaceable part. Keep prompts, evals and integrations independent of any one provider.
Models will sometimes produce unsafe, off-policy or malformed output. Guardrails catch it before it reaches a user or a downstream system.
Put guardrails in code at the boundary, not only in the prompt. A rule the model can ignore is a suggestion.
Models generate likely text, not checked facts. Hallucination can be reduced but not eliminated, so the system must be designed to catch it.
Ground answers in retrieved sources, ask for citations and check them. Design for the case where the model is wrong.
Some actions are too costly or too final to leave to a model. Approval is how the system stays useful without being trusted with everything.
Model approval as a durable wait: the run pauses, survives restarts and resumes exactly where it stopped.
Vector search misses exact terms such as product codes, names and clause numbers. Keyword search misses paraphrases. Together they cover each other.
Start with hybrid search and a reranker. Pure vector search is rarely the right default for business documents.
Agents retry, crash and resume. Without idempotency, a retry sends the email twice or pays the invoice twice.
Make every write action idempotent, with a key per step, before any agent is allowed to call it.
Inference is where the running cost and the latency of an AI system come from. Training happens once; inference happens on every request, forever.
Budget inference per task, not per token. A system that is cheap per call but makes nine calls per answer is not cheap.
A model that cannot read from or write to your systems produces suggestions someone has to copy by hand.
Integrate into the systems people already use. A new screen for the AI is usually a screen nobody opens.
Retrieval, model calls, tool calls and checks all spend from the same budget. Without one, each step looks fine alone and the whole answer is still too slow.
Set the budget from the user’s workflow first, then decide how many model calls fit inside it.
MCP lets one integration serve many AI applications, instead of a custom connector per app and per model.
MCP standardises the connection, not the trust. Permissions and effect classes still need designing for each tool.
Models are stateless. Any memory is something the system stores and reinserts, and stale or wrong memories quietly degrade answers.
Engineer memory explicitly: what is stored, for how long, who can see it and when it is forgotten.
Most requests in a real workload are routine. Sending all of them to the largest model pays frontier prices for work a small model does just as well.
Route on measured quality per task type, backed by an eval set, not on a guess about which requests look hard.
An AI system fails quietly: a slower answer, a worse answer, a bigger bill. Traces of every prompt, retrieval and tool call are how you find out which step changed.
Trace and cost every model call from the first day. Adding it after the first incident means debugging the incident blind.
A model shown a document will use it. If permissions are checked after retrieval, or not at all, the system leaks information in its answers.
Filter by permission before you rank. Access control belongs in the query, not in a prompt asking the model to be careful.
Plans make agent behaviour easier to inspect and to stop. Plans the agent never checks drift away from the goal.
Make plans visible and checkable, and put human review where steps are expensive or hard to undo.
In production, a prompt is code. A small wording change can alter behaviour across thousands of requests.
Version prompts, review them like code and never ship a prompt change without running the eval set.
Any document, email or web page a system reads can carry instructions. If the model can act on them with tools, the attacker can act too.
Treat it as an input trust problem: limit what untrusted content can trigger, and never rely on the model to refuse.
Retrieval-augmented generation (RAG) lets a model answer from your documents instead of from memory. Most wrong answers in a RAG system are retrieval failures, not model failures.
Evaluate retrieval on its own, before the model sees anything. If the right passage is not in the context, no prompt will fix it.
Other software can only act on output it can parse. Free text in the middle of a pipeline is where integrations break.
Validate every structured output at the boundary, and decide in advance what happens when validation fails.
Low temperature gives more repeatable answers; high temperature gives more varied ones. It does not make a model more or less truthful.
Keep temperature low for extraction, classification and tool calls. Test the setting with evals, not by feel.
Context limits, prices and speed are all counted in tokens. Some languages, codes and numbers take far more tokens than they look like they should.
Count tokens on real inputs early. Many cost and context surprises are tokenisation surprises.
Tools are how a model touches real systems. Every tool is also a new way for a mistake, or an attacker, to cause a real effect.
Give every tool a typed schema, a permission scope and an effect class: does it read, write or do something irreversible?
Attention is why models handle long-range meaning well, and why cost and memory grow as the context gets longer.
You do not need to build one to use one well, but knowing how attention scales explains most context-window trade-offs.
It is the lookup step behind most retrieval systems. It is fast and approximate, and it knows nothing about who is allowed to see what.
Treat the vector index as one retriever among several, not as the knowledge base itself.
Adoption depends on fitting the work people already do. The best model in the wrong place in the workflow goes unused.
Start from the job and the workflow, then decide where a model helps. Not the other way round.