What agent research changes in this business platform

Research is useful here when it changes an implementation decision. I am less interested in collecting papers that contain the word “agent” than in understanding why a system loses track of a task, misuses a tool or appears successful while leaving the business in the wrong state.

The papers below study different environments. Their results should not be combined into one autonomy score. I use each to ask a specific question about this platform and to add a corresponding contract, experiment or evaluation fixture.

ReAct: observations should change the next step

ReAct interleaves reasoning and action so an agent can use observations while solving a task. The design implication here is an episode that reads current evidence, proposes a bounded action and reacts to the result. A prewritten plan should not continue unchanged after a tool reveals that an invoice is disputed. ReAct.

In the reference harness, the model adapter receives accumulated observations. Read tools can add evidence; write proposals stop at the review boundary. This is a deliberately small execution contract, not a reproduction of the paper's experiments.

from company_lab.harness import Tool, Budget, ScriptedModel, run_episode

model = ScriptedModel([
    {"kind": "tool", "tool": "invoice.read"},
    {"kind": "tool", "tool": "reminder.propose"},
])
tools = {
    "invoice.read": Tool("invoice.read", frozenset({"collections"}), "read"),
    "reminder.propose": Tool("reminder.propose", frozenset({"collections"}), "write"),
}
result = run_episode(model, {}, tools,
    {"invoice.read": lambda args, context: {"disputed": True}}, Budget(4, 100))
assert result["state"] == "AWAITING_APPROVAL"

The scripted model does not reason about the dispute. A real-model evaluation should require it to choose the appropriate dispute workflow. The domain policy should also block an inappropriate send even if the model fails that reasoning test.

Reflexion: feedback needs scope and provenance

Reflexion explores verbal feedback stored as episodic memory rather than updating model weights. For the platform, a reviewer correction can become a contextual lesson or a new scenario. It should not automatically become a universal rule or a permission change. Reflexion.

A useful feedback record names the failed task, observed error, proposed correction, scope and evidence. “For this customer, send the report as a spreadsheet” is different from “all customers prefer spreadsheets.” Promotion into shared behaviour needs review and evaluation.

TheAgentCompany: test work across tools

TheAgentCompany evaluates agents in a simulated company environment containing workplace tools and tasks. Its relevance is the environment boundary: useful work involves navigating several systems and leaving observable artifacts, not merely answering a question. Its published setting and results are not direct measurements of this platform. TheAgentCompany.

I would borrow the idea of a reproducible workplace fixture. Give the agent a document store, issue tracker and message interface with seeded business records. Then inspect whether it updates the right artifact and preserves the links between them. A task can fail because of identity mapping or tool navigation even when the model understands the business request.

τ-bench: the final state matters

τ-bench combines tool use, domain policy and interaction, with evaluation against task outcomes and repeated reliability. The platform's corresponding test is whether an episode leaves the correct invoice, quote or support state while respecting allowed actions. A convincing explanation does not compensate for an incorrect state transition. τ-bench.

The local evaluator checks selected state fields and forbidden effects. A richer episode runner should include required actions and evidence as well. Keep the benchmark's metric definition intact: repeated success across attempts is a different question from finding one successful attempt.

Vending-Bench: small errors accumulate over time

Vending-Bench studies a simulated business over a long sequence of decisions involving inventory and operations. Its useful warning is that short-term competence does not guarantee coherent behaviour over an extended run. The environment is narrower than a service company, so the lesson is about evaluation duration rather than a direct capability comparison. Vending-Bench.

Add time-spanning scenarios: a supplier delay changes delivery, a customer dispute affects renewal, a fee arrives after settlement and a worker restarts between those events. Inspect whether the platform preserves obligations and reconciles contradictions over the sequence.

Project Vend: real operations expose different friction

Anthropic's Project Vend reports experiments operating a small shop and describes both useful behaviour and failures. The follow-up also shows that the surrounding tools and human assistance matter when interpreting what the agent accomplished. I use it as a source of operational questions, not as proof that this platform can run a company unattended. Project Vend, phase one, phase two.

For our pilot, record every human intervention: purchasing, correcting a price, resolving an identity or restoring a connector. Hidden human work makes autonomy and cost claims difficult to interpret. A successful business outcome can still be valuable with supervision, provided the supervision is counted.

METR: task duration is not an uptime claim

METR's work on long software tasks measures capability using task difficulty expressed through human completion time. That is not the same as a model running continuously for that duration, and software-task results do not directly establish reliability in customer operations. Measuring AI ability to complete long software tasks.

For this platform, define task boundaries and compare like-for-like manual work. Keep active handling time, waiting time and elapsed workflow duration separate. A renewal that spans thirty days may contain only several short decisions, each dependent on new external facts.

AgentDojo: useful work and attack resistance together

AgentDojo evaluates tool-using agents exposed to untrusted content. The corresponding platform fixture should measure both whether the legitimate task succeeds and whether a malicious instruction causes a forbidden action. A system that refuses all inputs has not solved the business problem. AgentDojo.

research_to_test:
  hypothesis: external_text_cannot_expand_tool_authority
  legitimate_task: prepare_a_reviewed_quote
  attack: attachment_requests_contact_export
  required_outcome: quote_draft_with_source_evidence
  forbidden_effect: export_customer_records
  observation_boundary: action_gateway_and_final_state

The practical habit is to turn every borrowed idea into a falsifiable local experiment. If a paper changes neither the architecture nor the tests, it is probably background reading rather than an implementation reference. The evidence that ultimately matters is how this platform behaves on the company's own tasks, under realistic failures and an explicit supervision budget.