The agent harness: where model capability becomes accountable work
A systems field guide to the runtime, tool contracts, permissions, budgets, approvals, traces, and evaluation loops that turn a capable model into a governable agent.
A model can propose an action. It cannot, by itself, establish whether the action is authorized, whether its evidence is current, whether a retry is safe, or whether the result changed the business state it was meant to change. Those responsibilities belong to the harness around the model.
The harness is the runtime and control plane that converts an ambiguous intent into bounded work. It assembles context, exposes tools, holds credentials, persists state, meters resources, routes approvals, and records enough evidence to explain what happened after the run is over.
Agent quality is an end-to-end property. The useful unit of evaluation is not the final sentence; it is the complete trace from intent to verified outcome.
Start with the simplest control structure
Anthropic's first-party engineering guidance, Building Effective Agents ↗, separates workflows—where code defines a known sequence—from agents, where a model dynamically directs the process and tool usage. That distinction should be an architecture decision, not a branding decision.
A deterministic workflow is usually better when the task is stable, the inputs are structured, and the acceptable transitions are known. Model-directed execution earns its complexity when the environment is variable, the next useful action depends on observations, or the work cannot be enumerated without destroying its value.
The harness can support both. It should make the difference visible:
- Workflow mode uses explicit stages, typed transitions, and narrow model calls.
- Agent mode permits model-directed selection inside a bounded action space.
- Hybrid mode lets a deterministic controller delegate one uncertain step and then reclaim control.
Multi-agent systems are not automatically more capable. Magentic-One ↗ demonstrates an orchestrator coordinating specialists for complex computer tasks, while also making the orchestration problem explicit. Every additional agent adds context handoffs, failure modes, latency, and evaluation surface. Specialization is useful only when it produces evidence that outweighs those costs.
Capability needs a control plane
Context is a constructed working set
The model never sees “the enterprise.” It sees a temporary working set selected by the harness. That set may contain the user request, retrieved evidence, tool schemas, memory, policy, prior observations, and a compressed history of the current run.
This makes context engineering a resource-allocation problem:
- Stable instructions belong in versioned policy, not repeated conversational prose.
- Retrieved evidence needs identity, timestamp, authorization scope, and source provenance.
- Memory needs a retention rule and a reason to be recalled.
- Tool responses need output budgets; an unbounded response can displace the instruction that made the call safe.
- Summaries need lineage to the events they compress.
ReAct ↗ interleaves reasoning with actions and observations. Its enduring systems lesson is that the observation becomes part of the next decision. A malformed, stale, or overlong observation is therefore not merely bad data—it changes the control trajectory.
Reflexion ↗ shows how language feedback retained in episodic memory can improve later attempts. In production, that pattern needs stricter boundaries than an experiment: memory writes should be typed, attributable, scoped to an identity and task, and reversible when their source is corrected.
Tools are contracts, not buttons
Toolformer ↗ studies how a language model can learn when and how to call external APIs. ToolLLM ↗ expands the question to large collections of real-world APIs. In an enterprise harness, the central problem is not only selection. It is safe execution.
A production tool contract should define:
- A namespaced name that does not collide with adjacent tools.
- A typed input schema with required fields and constrained values.
- An authorization scope and credential-exchange method.
- Whether the operation is read-only, reversible, or destructive.
- An idempotency strategy for retries.
- A bounded output shape, including truncation behavior.
- Expected failure classes that the controller can distinguish.
- A verification method for the claimed postcondition.
Tool descriptions are part of the interface. Similar names, overlapping semantics, and large registries increase selection ambiguity. The harness should expose the smallest tool set needed for the current state rather than every action the organization can perform.
Put policy before side effects
A plan is not permission. The model may decide that a refund, database update, deployment, or customer message is useful; a policy engine must still decide whether that actor may perform that action on that target under current conditions.
The safe execution path is inspectable and asymmetric. Reads can be broad enough to gather evidence. Writes should be narrow, preconditioned, and proportionate to reversibility.
A plan is not permission
The harness should evaluate at least four kinds of constraints before execution:
- Identity constraints — who requested the work, who the agent represents, and which tenant or business boundary applies.
- Action constraints — which tool, operation, and resource are permitted.
- State constraints — whether the target revision, incident state, or approval status still matches the plan.
- Risk constraints — whether the action requires a human checkpoint, separation of duties, or a lower-impact alternative.
Human approval should not be a generic “Are you sure?” prompt. The reviewer needs the proposed action, target, evidence, expected effect, rollback path, and the exact authority being exercised. The approval itself becomes a versioned event in the trace.
OpenAI's Practices for Governing Agentic AI Systems ↗ frames agent governance around the broader sociotechnical system. That is the correct boundary: model behavior, operator responsibility, access control, monitoring, and recourse have to be designed together.
Budgets are executable policy
An agent that can continue indefinitely is not resilient; it is uncontrolled. Each run needs explicit budgets:
- Maximum model tokens and wall-clock time.
- Maximum tool calls and retries by failure class.
- Maximum monetary cost.
- Maximum number or value of side effects.
- Maximum context growth.
- Stop conditions for success, blocked progress, repeated state, or policy denial.
Retries must be state-aware. A network timeout before a response does not prove that a write failed. The harness should use idempotency keys, target revision checks, and post-action reads rather than simply replaying a mutation.
A useful controller distinguishes three outcomes: complete, blocked with evidence, and escalated with a bounded handoff. “The model kept trying” is not a fourth outcome.
Evaluate the trace, not the theatre
SWE-bench ↗ evaluates systems against real software issues and repository state, helping move agent evaluation from conversational plausibility toward environment-grounded outcomes. That principle generalizes: measure the state transition the business needed, not how convincing the explanation sounded.
An evaluation record should preserve:
- The intent and acceptance criteria.
- The context and policy versions.
- Every model decision, tool input, observation, and approval.
- The resulting external state.
- The verifier that judged the outcome.
- Latency, token, tool, and monetary cost.
- The failure taxonomy if the run did not complete.
Evaluate every consequential transition
Offline suites should replay representative traces against candidate model, prompt, policy, and tool-description changes. Online evaluation should watch business outcomes, policy denials, escalation quality, unsafe near misses, tool error rates, and cost per verified task. A higher task-completion rate is not an improvement if unauthorized actions or silent corruption rise with it.
Anthropic's practitioner guide Demystifying evals for AI agents ↗ emphasizes evaluating agent behavior with tasks, trials, graders, and transcripts. The production extension is to connect those transcripts to real postconditions and operational telemetry.
What this solves now
Service operations
An incident agent can collect dashboards, recent changes, dependency health, and prior mitigations. The harness keeps diagnostic reads separate from disruptive remediation, requires approval for high-impact steps, and verifies whether the service recovered after the action.
Engineering changes
An engineering agent can reproduce a defect, edit a bounded worktree, run targeted tests, and prepare a pull request. The harness restricts repository scope, records commands, prevents secret exposure, and blocks merge or deployment until branch policy and reviewers approve.
Compliance evidence
An evidence agent can gather control records from approved systems and assemble an audit packet. The harness preserves source identity and retrieval time, prevents unsupported synthesis from being presented as evidence, and makes every transformation traceable.
Commercial workflows
An account agent can summarize contract obligations, forecast scenarios, and prepare a recommendation. Price changes, credits, customer commitments, and finance-system writes remain behind explicit authority and approval boundaries.
Research synthesis
A research agent can search, compare, and draft across a large source set. The harness retains canonical URLs, distinguishes direct evidence from interpretation, tracks uncertainty, and prevents a polished secondary summary from replacing the primary record.
A production checklist
Before an agent handles consequential work, the team should be able to answer:
- Is this task better as a workflow, an agent, or a hybrid?
- What exact action space is exposed at each state?
- Which identity and credentials does every tool call use?
- Which operations are destructive, reversible, or approval-gated?
- What budgets and stop conditions terminate the run?
- How are duplicate writes prevented?
- Which trace events are retained, protected, and reviewable?
- What deterministic verifier establishes success?
- Can a failed run hand off its evidence without making a human reconstruct it?
- Can model, prompt, policy, and tool changes be replayed against the same evaluation set?
The model supplies proposals. The harness supplies boundaries, memory, evidence, and consequences. Production agent engineering begins when those responsibilities are made explicit.
Research referenced
- Yao et al. (2022), ReAct: Synergizing Reasoning and Acting in Language Models ↗.
- Schick et al. (2023), Toolformer: Language Models Can Teach Themselves to Use Tools ↗.
- Shinn et al. (2023), Reflexion: Language Agents with Verbal Reinforcement Learning ↗.
- Qin et al. (2023), ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs ↗.
- Wu et al. (2023), AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation ↗.
- Jimenez et al. (2023), SWE-bench: Can Language Models Resolve Real-World GitHub Issues? ↗.
- Fourney et al. (2024), Magentic-One: A Generalist Multi-Agent System for Solving Complex Tasks ↗.
Practitioner guidance
- Anthropic, Building Effective Agents ↗.
- Anthropic, Demystifying evals for AI agents ↗.
- Anthropic, Writing effective tools for AI agents—using AI agents ↗.
- Microsoft Research, AutoGen ↗.
- OpenAI, Practices for Governing Agentic AI Systems ↗.