The next AI-agent test: what can it do, and what can it touch?
Why a capable model needs a carefully designed operating environment—and how to evaluate one without being an engineer.
CryptoXAI Editorial · 2026-09-08 · Source-based analysis, AI-assisted
A development worth following
Anthropic’s May 2026 engineering account of containment describes different environments for different products. Its discussion of Claude Code highlights the usefulness—and the responsibility—of access to a user’s filesystem, shell and network. This is an architectural issue that applies beyond one model vendor.
Meanwhile, Google’s 1 September roundup describes its August AI releases, including Gemini 3.7 Flash and Gemini 3.5 Transcribe. More capable and specialised components broaden what teams can build. Our interpretation is that product evaluation must now examine the surrounding workflow as carefully as the model announcement.
Read, propose, execute
Imagine an assistant reviewing an invoice. Reading the invoice, drafting an accounting entry and sending a payment are three distinct levels of authority. A useful product makes those transitions visible. The same principle applies to code: explaining a patch is different from modifying a repository, which is different from deploying it.
This three-level distinction is our evaluation framework. It is not a claim that every current tool exposes the same controls. Before adopting one, ask for a demonstration of the exact action you care about, including how it requests authority and how the action appears in the activity record.
- ReadInspect only the authorised inputs
- ProposeExplain the intended change
- ExecuteApply within granted permissions
- VerifyCheck outcome and retain an audit trail
What the demonstration should include
Ask the operator to supply an incomplete request, a conflicting instruction and an unavailable dependency. Does the system stop with a useful explanation, ask for the missing fact, or silently invent a successful outcome? The failure path often reveals more about operational quality than the polished example.
For a reversible action, ask how to undo it. For an irreversible one, ask what confirmation and limits apply. Check whether the system can distinguish content it is reading from instructions it is authorised to follow. None of these questions requires believing that a model can be made infallible.
Measure useful autonomy
Define a completed task in advance and count the interventions needed to reach it. Include review time, retries and abandoned attempts. A workflow that completes ten tasks with one inspection each may be more valuable than a spectacular autonomous run that cannot be reproduced.
For buyers, the next step is a small trial with bounded permissions and representative inputs. For builders, the opportunity is to turn model capability into a dependable experience: clear boundaries, visible progress and recoverable errors. More intelligence helps; it does not remove the need for product engineering.
Sources & method
Official sources reviewed for this edition. Recommendations and evaluation frameworks are CryptoXAI analysis. No hands-on benchmark results are claimed.