Benchmark · 2026-10-23 · GLM-5.3 Flash · 282 agent actions
Agents keep to a team’s rules more often with Claros.
Same model, same agent actions. Claros against the two ways teams hand an agent its rules today: the docs in its context, and RAG over them.
Read the whitepaper+16pts accuracy over the team’s docs in the agent’s context, on both sets
+24–29pts accuracy over RAG on the same docs
91%accuracy on the held-out set, 160 actions
Accuracy
The verdict on the action matches the label: breaks the rule, fine, or can’t tell.
Claros93%
Repo docs77%
RAG69%
Claros91%
Repo docs75%
RAG63%
Results
GLM-5.3 Flash, every condition on the same cases.
| Standard · 122 | Claros | Repo docs | RAG |
|---|---|---|---|
| Accuracy | 93% | 77% | 69% |
| Breaks answered as the team set | 94% | 73% | 70% |
| Can’t-tell cases called can’t tell | 80% | 7% | 17% |
| Fine actions held (lower is better) | 9% | 39% | 27% |
| Held out · 160 | Claros | Repo docs | RAG |
|---|---|---|---|
| Accuracy | 91% | 75% | 63% |
| Breaks answered as the team set | 87% | 58% | 54% |
| Can’t-tell cases called can’t tell | 64% | 19% | 17% |
| Fine actions held (lower is better) | 15% | 47% | 35% |
How it was run
- 282 agent actions in two frozen sets, each labelled breaks the rule, fine, or can’t tell.
- The held-out set was drafted by another model and frozen before any run. Claros was never tuned on it.
- Baselines: the repo’s docs in the agent’s context (~41k tokens), and RAG over them (top 5 passages).
- Claros gets the full workspace, all 30 rules, and routes each action itself. No rule is picked for it.
- Rules: 30 illustrative team rules; labels drafted by models, not owners.