Technology
How we approach autonomous-agent engineering.
A proposed architecture and a set of design principles. Everything on this page is a reference model: how we think a dependable agent system should be organized, not a description of a shipped product.
On this page
IntroductionExecution lifecycleReference architectureObjective interpretationPlanning and decompositionTool selection and executionContext and memoryVerification and evaluationSafety and controlEngineering principlesDiscuss a problemWhat an agent system needs beyond a model
A language model can propose the next step. A dependable agent system decides which proposals to accept, runs them under limits, and checks what happened.
We use “agent” in an operational sense: a system that pursues an objective through repeated cycles of planning, acting with tools and checking results. Seven concerns recur in every such system: interpreting the objective, planning, managing context, choosing and executing tools, verifying results, and enforcing safety and approval. The reference model below gives each a name, a responsibility and an interface so they can be designed, tested and replaced separately.
The agent execution lifecycle
A run is a loop, not a line. After a step executes, the system observes what actually happened and compares it with what it expected. An unexpected result can send the agent back to revise its plan; a check that fails against the acceptance criteria sends it back to retry the step. The loop ends in one of three ways: the criteria are met, a person has to decide, or the run stops safely.
- Criteria met
The outcome is reported together with the evidence that supports it.
- Uncertain or sensitive
The run pauses and a person decides, with the evidence in front of them.
- Unsafe or out of budget
The run stops safely and reports what was done, what was not, and why.
Reference architecture
Six components and one control layer. The Planning Engine and the Verification Engine are joined by a feedback loop; the Tool Execution Layer sits inside a permission boundary; Context and Memory is a shared bus every component reads and writes under retention rules. The Safety and Control Layer is deliberately drawn apart: it is not a feature of one component but a constraint on the whole.
Fig. 2Conceptual reference architecture: proposed design, not a representation of a verified production deployment.
Objective interpretation
Most agent failures begin before the first action, with a request that was ambiguous, under-specified or quietly out of scope. Objective interpretation converts a request into a structured objective, and it insists on one thing before planning starts: a definition of done that can be checked.
- Goal
- What outcome is wanted, in one sentence a person could confirm.
- Constraints
- What must not change, what may not be touched, budgets for cost and time.
- Scope
- The systems, data and tools the task may involve, and nothing beyond them.
- Acceptance criteria
- Checkable statements. If a criterion cannot be checked, that is itself reported.
- Open questions
- Ambiguities to resolve by asking, rather than by guessing.
- Risk class
- How much damage a wrong action could do, which sets the default level of autonomy.
Open question: how much clarification helps before it becomes an obstacle? Too few questions yields guesses; too many removes the value of delegation.
Planning and decomposition
A plan is the agent’s commitment about what to do and in what order. Making it explicit, as a stored graph of steps with dependencies rather than a sequence implied by a chain of calls, is what lets anyone review it and lets the system revise it deliberately.
Each step carries the capability it needs and its own checkable criterion. Steps that do not depend on one another can run in parallel. The plan is recorded, can be shown to a person for review before execution, and has limits on size and depth so that decomposition cannot run away.
Replanning triggers
- A step fails in a way retrying will not fix.
- A result contradicts an assumption the plan relied on.
- New information changes the objective’s constraints.
- A budget is being consumed faster than the plan allowed for.
Open questions
- How do you know a plan is adequate before running it?
- When should a plan be patched, and when replaced?
- How is a stale plan detected?
Tool selection and execution
A tool is a capability with a contract: what it takes, what it returns, what it can change and what it costs. Selecting a tool means matching a plan step to a contract the agent is permitted to use, then preparing the parameters. Only after a policy check does the call reach the execution runtime.
| Field | Why it exists |
|---|---|
| Typed inputs and outputs | Malformed calls can be rejected before they run, and results can be validated. |
| Scope | The data and systems the tool may touch for this task. |
| Side-effect class | Whether the tool only reads, changes something reversibly, changes something permanently, or acts outside the organization. |
| Cost and time limits | Keeps a loop of calls inside the run’s budget. |
| Idempotency | Decides whether a failed call can safely be repeated. |
| Failure modes | Lets the runtime classify an error instead of treating all errors alike. |
The execution runtime treats every step as a state with allowed transitions: pending, running, awaiting approval, succeeded, failed and retryable, failed and terminal, cancelled. Because transitions are defined, a failure leads somewhere specific. The runtime also checkpoints progress, enforces timeouts and budgets, and offers a stop control that does not depend on the agent’s cooperation.
Open question: where are rollbacks feasible once a tool has acted on an external system?
Context and memory
Context is a budget, and memory is a liability: each fact kept should earn its place and have a reason to expire. The reference model keeps three layers apart so that each can have its own scope and rules.
| Layer | Holds | Lifetime | Governing rule |
|---|---|---|---|
| Working context | What a component needs right now, assembled for one step. | One step or session | Fits a size budget; built from scoped views, not raw history. |
| Task state | The objective, the plan, results and artefacts so far. | The task | Tagged with provenance; deleted when the task closes unless policy says otherwise. |
| Retained memory | Anything kept beyond the task. | Defined by policy | Needs a stated purpose, an owner and an expiry. Sensitive fields are minimized or redacted. |
Open questions: what to summarize versus keep verbatim, and what an agent should forget.
Verification and evaluation
Done is a checked state. A system reports completion only after the evidence meets the criteria set at the start, and it stores that evidence with the run. The strongest check available for the task is used first.
| Check | Example | Strength | Caveat |
|---|---|---|---|
| Deterministic | Tests pass, schema validates, totals reconcile. | Repeatable and objective. | Only as good as the test or rule. |
| Reference-based | Output compared with a known-good result. | Catches drift from a baseline. | Needs a trustworthy reference. |
| Model-assisted | A second model judges output against a rubric. | Covers criteria that are hard to code. | Can share the actor’s blind spots; use alongside other checks. |
| Human review | A person inspects the evidence and decides. | Handles judgement and context. | Costs time; evidence must be presented so review is possible. |
Where feasible the verifier is kept separate from the actor, because a system tends to miss its own mistakes. Evaluating the system as a whole is a different problem: task-level success against acceptance criteria, step correctness, recovery from failure, cost and time, and how often a person had to intervene. Any one of these numbers can mislead on its own, which is why we treat evaluation method as a research topic. There are no results to report today.
Open question: who checks the checker?
Safety and control
Autonomy is granted in proportion to what a task can break and what can be checked, never by default. The control layer enforces that proportion through permissions, approvals, logging and stop controls. It reduces and exposes risk; it does not remove it.
| Action class | Examples | Default policy in the model |
|---|---|---|
| Read-only | Search, query, read a file. | Allowed within the task’s scope and logged. |
| Reversible write | Edit on a working branch, create a draft. | Allowed within limits, with the change recorded so it can be undone. |
| Irreversible | Delete, overwrite, deploy. | Requires human approval, with the evidence attached. |
| Externally visible | Send a message, change a record in another system. | Requires human approval, or stays out of scope. |
Controls the model calls for
- Permissions granted per task, narrowly, expiring with the task.
- Approval requests that carry evidence, not a bare yes or no.
- An append-only log of plans, calls, policy decisions, checks and approvals.
- Stop controls that do not rely on the agent’s cooperation.
- Data handling rules that also cover the traces themselves.
Open question: how do we design approvals that people can actually review well?
Engineering principles
Ten principles we design to. They are commitments about how we work, and none is a claim about results.
- Define done first. An objective carries acceptance criteria before work starts, so completion is checked rather than asserted.
- Plans are artefacts. A plan is stored in a structured, reviewable form and revised deliberately, not implied by a chain of calls.
- Tools are contracts. Each tool declares typed inputs and outputs, scope, side-effect class, cost and failure modes.
- Least privilege, by task. Access is granted per task, narrowly, and expires with it.
- Verify before reporting. A result is reported complete only after checks against the criteria.
- Separate actor and verifier where feasible. Independent checks catch errors the actor cannot see.
- Design for failure. Classify errors, bound retries, and stop safely with a report a person can use.
- Make runs observable. Plans, calls, decisions and checks leave an inspectable trace.
- Keep people in charge of consequence. Irreversible or externally visible actions need approval.
- State maturity plainly. Label every capability as planned, research, prototype or verified.
Contact
Discuss an agent architecture problem.
If you are designing, evaluating or questioning an agent system, tell us what you are working on.