Agent Reliability Study: Audit and OpenCode Tests
Our June 2026 homelab study provides useful examples of agent failures that require artifact verification. Its scoring, arithmetic, isolation labels, and scope need corrections before reuse as current operational guidance. This page separates the historical audit from new OpenCode qualification.
The original Silent failures, sandbox tiers, and reliability in multi-agent AI coding infrastructure remains available in Outline; access may require sign-in. Audit date: 2026-10-20. The checked Git snapshot is 9fe844d of the research repository. It contains v6.4; the supplied Outline text includes a later v6.5/E28 addendum whose raw data was not available in that snapshot.
What the historical evidence supports
Section titled “What the historical evidence supports”The durable lesson is to check the requested artifact and its behavior, alongside process status and tool events. A confident answer, HTTP success, or CLI exit zero is insufficient. The old numerical rates describe particular versions, prompts, gateways, and scoring rules; they are not current product rankings.
The audit reads recorded CSVs and generated files without executing historical model output or modifying the original observations.
| Study claim | Rechecked evidence | Correction |
|---|---|---|
| E13: about 22% of calls hang | 313 SUCCESS, 52 HANG, 35 NO_FILE out of 400 | 21.75% total failures; 13% recorded timeouts; 8.75% missing files. A timeout does not establish its cause. |
| E13: K=2 achieves 96.5% | 193/200 pairs pass the implemented scorer | This count is reproducible; Wilson 95% CI is 92.95–98.29%. |
| E13: the five-line contract is scored | The runner checks existence, syntax, and stdout, omitting line count, shebang, and comment | Adding static format checks to recorded successes leaves 263/400 calls and 180/200 pairs. This is a format-filtered historical count, not a fresh execution result. |
| E13: first success cancels its sibling | The runner waits for both children | Redundant final outcomes were measured; cancellation and first-success latency were not. |
| E18: retry costs about $0.012 per success | Recorded arm B total $0.07346 / 11 successes | $0.006678 per success already accounts for retries. Dividing by success probability again double-counts them. Vibe’s costs remain transcript-based estimates. |
| E18: all required format constraints are enforced | Its scorer also omits the format requirements | Recorded-success plus static-format checks gives 18/20 for refinement and 6/20 for one-shot, versus legacy 19/20 and 11/20. The arms also use different harnesses. |
| E16: a test of LXC isolation | Its runner explicitly uses a remote host account without creating an LXC | It tests local versus remote-host permission behavior. It cannot support an LXC-specific conclusion. |
| Appendix: 931 rows in 17 CSVs | The listed files contain 1,371 rows across 22 CSVs | Rows mix pairs, single calls, and multi-call chains. The approximate invocation and spend totals need a reconciled ledger. |
For independent attempts with all-failure probability q, the probability of at least one success in two attempts is 1 − q². E13’s legacy endpoint gives 1 − (87/400)² = 95.27%, close to its observed 96.5%. That agreement is compatible with independence; it does not prove independence or cover shared provider outages.
The supplied v6.5 already notes invocation/regime contamination and the need for an E29 rerun. Retain that calibration instead of interpreting the older failure rate as an intrinsic current-model hang probability.
Refinement and retries also answer different questions. The historical second step often recovered a missing file. A controlled comparison must keep the model, harness, scorer, prompt budget, and task set consistent while comparing one attempt, conditional retry, and review-and-fix.
Replace the strict sandbox ranking with explicit boundaries
Section titled “Replace the strict sandbox ranking with explicit boundaries”Location, kernel separation, credentials, mounts, network access, and orchestration are separate dimensions.
| Arrangement | Execution boundary | What still needs verification |
|---|---|---|
| Process sandbox on the Mac | Kernel-enforced process policy | Allowed paths, sockets, subprocesses, credentials, and egress |
| Local Linux VM | Guest kernel separated from the Mac | Shared folders, credentials, host services, and network policy |
| LXC on a remote server | Namespaces and policy sharing that server’s kernel | Host/container escape surface, mounts, credentials, egress, and budgets |
| Several workers | An orchestration topology using the chosen boundaries | Each worker’s access, shared resources, cancellation, and acceptance |
LXC does not provide a private kernel for each container. Apple container runs lightweight Linux VMs. Fan-out does not inherently increase isolation. Missing host paths and distinct output words are useful checks, but do not establish resistance to exfiltration or attempted cross-worker reads.
The paper’s old Docker status is historical. Current Docker Sandboxes documentation describes sbx and microVM isolation; this audit did not qualify that product locally. Planned budget keys or network restrictions must not be described as enabled controls.
Current OpenCode qualification
Section titled “Current OpenCode qualification”The new protocol was committed before the main run: native OpenCode 2.1.0, Go Plus, explicit opencode-go/qwen3.7-plus and opencode-go/glm-5.3-flash, two concurrent jobs maximum. These are fresh current measurements, not an exact replication of the old Vibe invocation stack.
Each call uses a private standalone server, fresh session, closed stdin, warming disabled, and a custom agent restricted to two synthetic fixture files. Agent shell, external reads, web, skills, MCP, and delegation are denied. Separate Seatbelt validators execute generated code with no network or filesystem writes, restricted content reads, minimal environment, resource limits, and a deadline. Positive and negative controls verify the scorer before inference. Agent tool permissions themselves are not OS isolation.
The main run retained 81 CLI attempts and 211 model-turn records. All processes exited zero; one generated four-line artifact failed the strict scorer. There were no 90-second deadlines. The two artifact-repair families passed 10/10 each. The environment was an arm64 Mac on macOS 27.0.1, with a deliberately minimal synthetic instruction profile.
| Model | Task/stage | Accepted | Wilson 95% CI | Median CLI seconds |
|---|---|---|---|---|
| glm-5.3-flash | Behavior-focused script | 10/10 | 72.2–100.0% | 3.20 |
| glm-5.3-flash | Five-line script | 9/10 | 59.6–98.2% | 10.32 |
| glm-5.3-flash | Review-and-fix step | 10/10 | 72.2–100.0% | 7.96 |
| glm-5.3-flash | Conditional fresh retry | 1/1 | 20.7–100.0% | 6.97 |
| qwen3.7-plus | Behavior-focused script | 10/10 | 72.2–100.0% | 10.03 |
| qwen3.7-plus | Defensive SQLite repair | 10/10 | 72.2–100.0% | 15.57 |
| qwen3.7-plus | Five-line script | 10/10 | 72.2–100.0% | 20.69 |
| qwen3.7-plus | Seeded nested-dict repair | 10/10 | 72.2–100.0% | 21.53 |
| qwen3.7-plus | Review-and-fix step | 10/10 | 72.2–100.0% | 14.25 |
Both refinement and the fresh retry corrected GLM’s one miss. Across its ten initial five-line tasks, conditional retry reached 10/10 accepted results using 11 calls, while always refining used 20 calls for 10/10. Qwen’s ten cases passed initially and gained no acceptance improvement from a second pass. Neither model regressed in refinement. These paired strategies reuse the initial observations and run follow-ups later; they are not independently randomized arms.
The gateway-reported cost for the main run was $0.145976, rather than an extra cash charge to the Go subscriber. Cache-read counters appeared on both models. No controlled cache-on/off test was performed. The 211 turn records are not an independently audited HTTP-request or account-billing count. Exploratory preflight probes and study-review calls are kept separate from these measurements.
The additional study reviews also illustrate why a deadline is not a diagnosis: Qwen wrote its review file but reached the 90-second process limit; GLM completed. Both supplied useful leads and still needed verification. One review reproduced an incorrect variance expression, and another suggested an insufficient kernel check. Model agreement did not determine the final audit conclusions.
Two preflight lessons
Section titled “Two preflight lessons”Check the logical directory as well as the process cwd. In the Python launcher, changing cwd while retaining an inherited PWD made this OpenCode installation select the wrong location. The custom agent was unresolved and a default-agent write landed outside the fixture. Aligning both restored the intended location:
env = os.environ.copy()env["PWD"] = str(fixture.resolve())# Launch with cwd=fixture and this child environment.Exit zero still needs interpretation. An intentional permission-denial canary under --auto returned exit zero, a structured Permission denied: edit, and no file. Explicit deny remained active and the model accurately reported the denial. This was a visible refusal, not reproduction of the old Claude env-scrub bug. The actual artifact remains the acceptance criterion.
What these tests cannot establish
Section titled “What these tests cannot establish”Ten trials per cell cannot establish rare-failure rates: even 10/10 has a Wilson 95% interval of about 72.2–100%. Small fixtures do not establish repository-scale coding quality, mixed-model child-agent reliability, long-session compaction, credential renewal, safe autonomous access, or superiority over Claude Code/Codex. A gateway-reported model ID is useful provenance, not independent attestation of underlying weights.
Cache counters show provider-reported reuse. Without a randomized cached/uncached comparison they do not establish a latency or cost saving. Go’s reported token cost is not the subscriber’s marginal cash bill; track quota and accepted work separately.
Daily-configuration follow-up (2026-10-21)
Section titled “Daily-configuration follow-up (2026-10-21)”A separate local setup follow-up installed the portable core, repaired shared skills, native Go profiles and bounded launcher. Qwen’s native build completed a read/edit/test canary, and a Qwen parent launched two GLM children that identified a seeded regression without changing files. Native permission checks included allowed read/glob controls and denied edit, shell, recursive dispatch, external access, credentials, grep, Code Mode and an unknown MCP action. The refreshed reliability skill also loaded and correctly distinguished missing output from accepted work and quiet stdout from a diagnosed hang.
See the versioned local receipt and installed workflow. These small canaries are not part of the 81-attempt benchmark and do not establish general mixed-model reliability or autonomous Proxmox qualification. Two required independent sequential stages compose as p × q; 1 − (1 − p)(1 − q) describes alternative attempts, not a generation-and-refinement pipeline. Conditional recovery needs its own measurements.
Refusal observations need narrower claims
Section titled “Refusal observations need narrower claims”The June corpus recorded different refusal rates across its configured surfaces. That does not establish a permanently unrestricted provider, or that one vendor universally refuses ordinary defensive work. Zero refusals in 60 trials has a Wilson upper bound of about 6.0%. Pin the model, account, route, date, prompt, and scorer when repeating those measurements.
Black-box refusal text does not locate the classifier inside a vendor’s infrastructure. One synthesis failure and one rejected verification application remain observations with limited scope; they do not prove a general context-scanning mechanism, automated rejection, or vendor strategy. Select a suitable model for legitimate authorized work and respect its provider’s policies. OpenCode tool approvals do not remove model response policies.
Related resources
Section titled “Related resources”-
Detailed audit and correction plan — research repository access required
-
Recorded results, protocol, and artifact map — research repository access required
Orchestration evaluation
Section titled “Orchestration evaluation”Use AI Agent Orchestration for the platform comparison and a proposed pilot that tests interruption, duplicate delivery, false completion, budget limits, and integration. Its proposed rubric is separate from measured results.