Abstract
Consider: a company commits to mitigating targeted harassment. Three teams — Trust & Safety, fairness research, and red teaming — are each responsible for operationalizing this commitment. Yet none of their evaluations trace directly to the policy's harm definitions, and results rarely feed back into deployment decisions.
This gap is increasingly prevalent for AI-centric products. Unlike traditional software, AI systems produce emergent behaviors difficult to anticipate in policy language and harder to capture with static evaluations. The result is a sociotechnical gap: safety policies describe intentions, while evaluation pipelines operate on different assumptions, owned by different teams, with limited traceability between them.
Using a system safety lens, participants work in cross-functional groups to (1) draw the control structure that operationalizes a real policy commitment, (2) identify where policy-to-evaluation gaps live, and (3) generate the research questions and tools that could close them.
Agenda
Six groups of five. Two facilitators roaming; each covers three groups.
- 010–30 min
Introduction & lightning talk
Shalaleh & Renee
Welcome and overview; state of policy and evaluations (Renee); control-structure terminology with a quick worked example (Shalaleh); Q&A.
- 0230–60 min
Activity 1 — Connecting policy to evaluations
Shalaleh
Groups identify stakeholders and draw the control structure for the policy on their table (worksheet steps 1–4).
- 0360–70 min
Share back
Shalaleh
Six groups × 1–2 min: each group presents its policy, control structure, and main observations.
- 0470–100 min
Activity 2 — Identifying & resolving gaps
Renee
Groups locate policy-to-evaluation gaps, brainstorm ways to close them, and name the research questions or tools that would help (worksheet steps 5–8).
- 05100–110 min
Share back
Renee
Six groups × 1 min: each reports one gap-closing strategy and a key research question or tool they surfaced.
- 06110–120 min
Debrief & conclude
Shalaleh & Renee
Connect findings from Activity 1 and 2, share next steps, point to workshop resources, and invite participants to stay connected.
What each activity produces
Activity 1 · 30 min
Connecting policy to evaluations
Groups work through steps 1–4 of the worksheet: identify the actors, establish power dynamics, and map control actions and feedback into a drawn control structure.
Activity 2 · 30 min
Identifying & resolving gaps
Groups work through steps 5–8: connect evaluations back to the policy, name the gaps, prioritize them, and propose closures — including research questions and tools that would help.
The worksheet, at a glance
Open full size ↗The full 8-step map groups take through in the room — from choosing a policy to closing evaluation gaps.
Who this is for
Trust & Safety leaders, operations managers, policy professionals, ML engineers, evaluation researchers, and product managers working on AI systems — especially those responsible for implementing or overseeing safety policies who need greater clarity on how those commitments translate into technical and operational controls.
Ready to map?
The worksheet works offline. Your draft saves locally and you can export to Markdown or print as PDF.