Evidence confidence
low. This source suggests a useful experiment but does not establish a reliable benefit in a real agent workflow.
What was observed
Check whether short tool evidence supports or explicitly contradicts a claim.
12 authored cases, 24 labels; batch versus two sequential SDK requests; alternating order, no cache or retries. Labels and corpus hash precede the first run. Live Jev via Vercel Gateway.
Baseline
The same questions in separate sequential requests; no expensive host-model comparison.
Finding
Batching reduced calls, input tokens and median case latency, with one additional incorrect answer in this small run.
- Batch: 23/24 labels correct; 12 calls; 4,113 input tokens; case p50 256ms, p95 535ms.
- Separate: 24/24 correct; 24 calls; 7,530 input tokens; case p50 537ms, p95 685ms.
- Both modes flagged 3/24 answers for review; zero failed requests.
What the result does not establish
Tiny synthetic diagnostic, not a held-out production benchmark. No host-agent tool overhead, final task success or invoice savings measured. One run is not a stable latency distribution.
What we would test in System One
Batch independent checks when evidence is shared, but evaluate quality as well as request count. Preserve uncertainty and inspect contradictions rather than blindly accepting the cheaper configuration.
This recommendation is our interpretation of the study. Related research does not establish the quality of every Engine recipe.
Try a related workflow
Primary sources
Read this record in System One Bench. Source commits are pinned where available. Review dates describe our inspection, not the original run date.
Metric definitions and review method · Submit a correction or new result