Scenario contributor
Read an assigned policy, identify operational rules, and convert them into agent tasks with enough context and observable evidence to evaluate consistently.
Discuss roleContribute
Build policy-grounded evaluation scenarios.
PiBench is inviting contributors for one focused need: converting real policy rules into realistic, reviewable dataset scenarios for language-interactive agents.
Current opening
Scope is intentionally narrow.
This opening is for dataset construction, not frontend, infrastructure, model training, or general paper contributions.
Policy areas
Three real-world policy documents.
| Domain | Policy focus | Scenario opportunity |
|---|---|---|
| Finance | FINRA anti-money-laundering guidance | Financial operations, required controls, escalation, and documentation. |
| Aviation | EU261 passenger-rights regulation | Disruption handling, assistance, reimbursement, compensation, and obligations. |
| Consumer protection | FTC Telemarketing Sales Rule | Sales-call conduct, disclosures, consent, prohibited practices, and records. |
The work
What contributors produce.
Policy basis
A precise connection to the clause, condition, exception, or obligation.
Operational context
A plausible role, request, relevant facts, and environment state.
Scenario interaction
A realistic task flow, including conversation only when needed.
Compliant behavior
A clear account of what the agent should do, avoid, or escalate.
Evaluation evidence
Observable conditions distinguishing compliance from plausible failure.
Quality standard
A scenario is not a policy question.
Insufficient
- A generic question asking what a rule says
- No traceable policy basis
- An expected answer that cannot be evaluated consistently
- An artificial edge case without a plausible role
Contribution-ready
- A real task governed by a specific rule
- Enough facts and state to resolve applicability
- A clear compliant path and plausible violation path
- Observable evidence across relevant episode surfaces
Contribution process
Review is part of dataset construction.
Contributors receive a policy area and guidance, submit drafts, and revise them through policy-grounding and evaluation-readiness review.