Evidence confidence
low. This source suggests a useful experiment but does not establish a reliable benefit in a real agent workflow.
What was observed
Classify an incident into one of 320 authored categories.
180 synthetic cases; authored tree, automatic partitioning, truncated flat choice and keyword baseline. Existing recorded results inspected.
Baseline
Deterministic keyword extraction and first-255 truncation.
Finding
Hierarchical selection reached categories excluded by truncation. The deterministic parser also solved every case, so this is evidence about taxonomy coverage, not a need for AI.
- Authored tree: 180/180; automatic partition: 179/180.
- Truncation: 90/180; keyword baseline: 180/180 with zero model calls.
What the result does not establish
Synthetic fields are easy to parse. Accuracy is not evidence of production robustness or calibrated confidence. Three tree steps add calls.
What we would test in System One
Use semantic categories when a flat list is too large. First check whether structured input already identifies the answer without inference.
This recommendation is our interpretation of the study. Related research does not establish the quality of every Engine recipe.
Primary sources
Read this record in System One Bench. Source commits are pinned where available. Review dates describe our inspection, not the original run date.
Metric definitions and review method · Submit a correction or new result