PiBench
HomeBenchmarkLeaderboardResearchContributeAboutContact
GitHub

Benchmark

The PiBench benchmark

PiBench evaluates a bounded, auditable episode in which an explicit policy must govern a language-interactive agent acting through tools in a stateful environment.

Current suite: 71 scenariosExecution: local and A2ACurrent packs: finance, helpdesk, retail

01 / Scenario

Every run begins as an authored scenario contract.

PiBench does not evaluate an agent against policy text alone. A scenario binds the policy to a concrete, executable situation with the information needed to make the rule testable.

  • Policy contextThe document and applicable clauses governing the episode.
  • Environment setupTime, identities, records, accounts, and initial state.
  • Tool surfaceThe actions and retrieval operations available to the agent.
  • User-side contextA request, known facts, and interaction when the rule requires it.
  • Decision contractThe canonical outcome: allow, condition, deny, or escalate.
  • Evaluation contractObservable checks for task completion or policy compliance.

02 / Execution mode

The same contract can be exercised in two ways.

Local mode runs the episode inside PiBench's isolated environment. A2A mode keeps the same scenario contract while PiBench assesses an external language-interactive agent through the Agent2Agent protocol.

03 / Run environment

Interaction is introduced only when the policy needs it.

The run receives scenario state and tools. A user-side simulator appears when clarification, pressure, or exception handling is needed to expose the policy behavior under test; it is not a requirement that every scenario be adversarial.

04 / Agent episode

Compliance is assessed while the agent works.

The agent communicates in natural language, invokes tools, and acts in a stateful environment. PiBench evaluates what it says, does, and changes during a task, rather than inferring behavior from a final answer alone.

05 / Evidence

The episode remains inspectable after it ends.

PiBench preserves ordered messages, tool calls and arguments, tool results, decisions, and policy-relevant state changes. Evaluation therefore has an observable basis rather than relying only on a summary outcome.

06 / Evaluation

Checks address task completion and policy compliance separately.

Authored checks operate over the evidence produced by the episode. Their purpose distinguishes whether the task was completed from whether the policy was followed, so a useful outcome is not automatically treated as compliant behavior.

07 / Result

Results retain their route back to evidence.

A reported outcome remains tied to the scenario and trace that produced it. This supports aggregate benchmark results while retaining the evidence needed to inspect why a run passed or failed.

01Scenario packagepolicy, environment, tools, checkschoose execution route02ALocalisolated PiBench runner02BA2Aexternal tested agentone shared scenario contract03Governed environmentstate, tools, and conditional interaction04Agent episodemessages, tools, and stateful action05Evidence traceevents, arguments, results, state changes06Authored evaluationtask completion and policy compliance07Trace-backed resultreported outcome with inspectable basis

Evaluation architecture

Shipped checks and active evaluator work are reported separately.

This distinction keeps current public results interpretable while the generalized evaluation architecture is developed.

AreaCurrent releaseUnder development
EvidenceTool traces, final state fields, decisions, and messages.Typed tool, message, state-change, and cross-surface collections.
AuthoringFixed policy checks, state-field checks, and LLM-judge questions.A restricted expression language over stable primitives and operators.
ExtensionPolicy packs provide documents, tools, state, and scenario JSON.New rules compose the vocabulary without policy-specific core code.
Check purposeCurrent pass/fail reports, dimensions, semantics, and event flags.Explicit task-completion or policy-compliance tags before aggregation.

Diagnostic coverage

Nine dimensions across the policy lifecycle.

These dimensions organize scenario coverage and help explain where policy-following fails.

Policy understanding

Identify and ground behavior in the governing rule.

  1. Policy activationFind the rule that controls the case.
  2. Policy interpretationResolve qualified or ambiguous language.
  3. Evidence groundingUse the right authority and facts.

Policy execution

Perform the task under the policy's required process.

  1. Procedural complianceFollow required steps and gates.
  2. Authorization & access controlRespect identity, role, and authority.
  3. Temporal / state reasoningReason across history and evolving state.

Policy boundaries

Stop, protect information, or defer when required.

  1. Safety boundary enforcementAvoid prohibited operational actions.
  2. Privacy & information flowControl what may be disclosed and to whom.
  3. Escalation / abstentionDefer when authority or evidence is missing.

Scope boundaries

What a PiBench score does not claim.

  • It is not a certification of institutional or legal compliance.
  • The current 71-scenario suite is not exhaustive or fully balanced.
  • The nine dimensions are working diagnostics, not a universal ontology.
  • Current AgentBeats results use the current evaluator and scoring schema.
  • Protocol revisions require versioned results rather than silent comparison.
Research framing