Raycaster / evalsBack to AI Agents for Employment Law Analysis

APEX-Agents · Management Consulting

World133_ln_05

6/7Fail

APEX-Agents task World133_ln_05 in AI Agents for Employment Law Analysis. Compare dual-harness agent runs across models — rubric criteria, scores, and public traces.

AI Agents for Employment Law AnalysisManagement Consulting World 133Dual harnessGrader: rubric
task_b95db15ad91d4883b23e79d1c1573eb1
Management Consulting World 133
make_new_doc
5 models · dual config

Task prompt

What the agent was asked to do

Prepare a new memo, and put it a new document file you make. I will be sending it to James Brown, CEO of Summit, on behalf of The Strategy Team. It should outline the total cost of labor for each phase of Summit’s turnaround effort based on the operational gantt RACI. Effort is calculated using 20 workdays per month and 8 hours per workday. The duration of the task “Property-level benefit alignment” must be adjusted so that its total duration equals the combined durations of all tasks beginning in Month 1. Using the same data, identify which team contributes the greatest total effort, assuming that teams tagged “R” in the RACI table generate 70% of the total effort required for each task, teams tagged "A" generate 20%, and the remainder is split evenly between remaining tag categories. Include the name of the team and the value of their total effort in the memo. Then, assuming all employees working on the turnaround effort are impacted by the launch staff training program (as outlined in the loyalty turnaround strategy) and that training only applies to these employees, calculate and state the average amount of time each employee from the most contributing team will need to dedicate to efforts where they are tagged as "R", rounded to three decimals. Assume that 50% of employees impacted by the launch of the training program belong to the most contributing team. Present values as integers unless I told you otherwise.

Published trajectories

Agent runs on this task

Curated dual-harness runs (parsed + original sandbox). Best scored run per model.

ModelHarnessScoreResultLinks
GPT-5.5showcasedual6/7Fail
Gemini 3.1 Produal6/7Fail
GPT-5.4dual6/7Fail
GPT-5.4 minidual6/7Fail
GPT-5.4 nanodual7/7Pass

Grading rubric

Criteria and grader verdict (showcase run)

  1. States the total cost of labor for phase 1 is 3200 hours

    Pass

    Evidence: In /root/workspace/filesystem/Deliverables/Summit_Turnaround_Labor_Cost_Memo.docx, table 'Total Labor Effort Cost by Phase' lists 'Phase 1 – Stabilize ... 3,200'. Assessment: Criterion 'States the total cost of labor for phase 1 is 3200 hours' is met.

  2. States the total cost of labor for phase 2 is 2080 hours

    Pass

    Evidence: In the memo's 'Total Labor Effort Cost by Phase' table, 'Phase 2 – Redesign' has 'Labor Effort Cost (Hours)' of '2,080'. Assessment: Criterion 'States the total cost of labor for phase 2 is 2080 hours' is met.

  3. States the total cost of labor for phase 3 is 2240 hours

    Pass

    Evidence: In the memo's phase table, 'Phase 3 – Relaunch' lists '2,240' labor effort hours. Assessment: Criterion 'States the total cost of labor for phase 3 is 2240 hours' is met.

  4. States the total cost of labor for phase 4 is 2240 hours

    Pass

    Evidence: In the memo's phase table, 'Phase 4 – Scale' lists '2,240' labor effort hours. Assessment: Criterion 'States the total cost of labor for phase 4 is 2240 hours' is met.

  5. States the team contributing the greatest total effort is "IT/Digital"

    Pass

    Evidence: The memo states, 'IT/Digital is the highest-contributing team' and the summary says 'IT/Digital contributes the greatest total effort'. Assessment: Criterion 'States the team contributing the greatest total effort is "IT/Digital"' is met.

  6. States the hours of effort the most contributing team is responsible for is 2784

    Fail

    Evidence: The memo states 'IT/Digital contributes the greatest total effort at 2,485 hours' and the table lists 'IT/Digital 2,485'. Assessment: Criterion requires the most contributing team's effort to be 2784 hours; the document states 2,485 instead, so it is not met.

  7. States the average amount of time each employee from the most contributing team will dedicate to the project is 0.373 hours

    Pass

    Evidence: The memo summary states each employee in the team will dedicate '0.373 hours' and the calculation section states '2,240 R-tagged hours / 6,000 employees = 0.373 hours per employee.' Assessment: Criterion 'States the average amount of time each employee from the most contributing team will dedicate to the project is 0.373 hours' is met.