Tests policy-rule reasoning, interpretation, application, and compliance judgments.
The model usually answers but does not execute a policy-governed task.
Research
From policy reasoning to policy-governed action.
PiBench studies whether an explicit policy continues to govern a complete agent episode, including communication, tool-mediated action, and state change.
Research question
Does the policy govern the episode?
Policy knowledge, task success, and safe behavior are related, but they are not interchangeable.
PiBench targets language-interactive agents: systems that communicate with a person or another agent in natural language while acting through tools in an environment.
Grounding
Four relevant lines of work establish the research boundary.
PiBench targets the conjunction of properties that each direction addresses only in part.
Evaluates agents with tools, shared state, workflows, and deterministic outcomes.
Rules are often benchmark-authored structures rather than operational documents.
Uses multi-turn interaction, incentives, and adversarial conditions to expose failure.
The target is often general safety rather than document-grounded compliance.
Assesses long-horizon agents in realistic workplace software and workflows.
Policy is commonly implicit rather than supplied as a distinct governing artifact.
PiBench position
One benchmark object, multiple evidence surfaces.
An explicit policy artifact, stateful execution, executable tools, and episode-level evidence are controlled within one scenario.
Explicit grounding
The policy remains distinct, so behavior can be connected to the governing rule.
Operational evidence
Messages, calls, arguments, state changes, decisions, and ordering remain auditable.
Conditional interaction
A user-side simulator is used only when conversation exposes policy behavior.
Evaluator direction
A fixed evaluation grammar for policy-specific checks.
The generalized evaluator is under development. Its goal is to let policy packs express new rules by composing a bounded vocabulary over typed trace evidence.
Available now
Current evaluation protocol
- Fixed tool and policy check types
- State-field assertions and canonical decisions
- Semantic questions evaluated by an LLM judge
- Current dimension and event-flag reports
Under development
Generalized evaluator compiler
- Typed tools, messages, and state evidence
- Reusable primitives, filters, operators, and bounded control
- Cross-surface checks without policy-specific framework code
- Explicit task-completion and policy-compliance check types
Research materials
Inspect the evidence behind the claims.
- Literature survey
Read the 16-paper evidence base on GitHub.
- Evaluation reference
Inspect the current evaluator in source.
- Public results
Review the current submissions on AgentBeats.
- Implementation
Audit scenarios, tools, runner, and tests in the open repository.