Raycaster / evalsBack to AI Agents for Healthcare and Senior Living Legal Risk

APEX-Agents · Law

Law433_mk_02

5/7Fail

APEX-Agents task Law433_mk_02 in AI Agents for Healthcare and Senior Living Legal Risk. Compare dual-harness agent runs across models — rubric criteria, scores, and public traces.

AI Agents for Healthcare and Senior Living Legal RiskLaw World 433Dual harnessGrader: rubric
task_3742720383624ce58ef4d751f8836392
Law World 433
edit_existing_doc
7 models · dual config

Task prompt

What the agent was asked to do

Can you edit the existing Grove Residency Agreement for Thomas Whitaker to make a revised residency agreement template that we can use for new residents. Please also update/add to the last sentence of the paragraph J in section IV of the agreement based on the 2025 Grove internal operations manual. Thanks.

Published trajectories

Agent runs on this task

Curated dual-harness runs (parsed + original sandbox). Best scored run per model.

ModelHarnessScoreResultLinks
GPT-5.5showcasedual5/7Fail
fireworks models Kimi K2dual5/7Fail
Gemini 3 Flashdual5/7Fail
Gemini 3.1 Produal3/7Fail
GPT-5.4dual6/7Fail
GPT-5.4 minidual4/7Fail
GPT-5.4 nanodual5/7Fail

Grading rubric

Criteria and grader verdict (showcase run)

  1. Removes any reference to "Thomas Whitaker"

    Pass

    Evidence: In `/root/workspace/filesystem/Magnolia Documents/Contracts/Revised Grove Residency Agreement Template 2025.docx`, the opening names the resident as a blank: `______________________________ ("Resident")`; signature block has `Print Name: ____________________________`. No `Thomas Whitaker`, `Thomas`, or `Whitaker` appears in the viewed text. Assessment: Criterion requires removing any reference to "Thomas Whitaker"; pass because the resident-specific name was replaced with blanks and no reference remains in the artifact text.

  2. Removes any reference to "Unit Number 111"

    Pass

    Evidence: In the agreement, Section I.A states `Resident has selected Unit Number ________ (the "Unit")`. No `Unit Number 111` appears in the viewed document text. Assessment: Criterion requires removing any reference to "Unit Number 111"; pass because the unit number is templated as a blank and the specific unit reference is absent.

  3. Removes any reference to the date August 1, 2023

    Pass

    Evidence: The opening date is blank: `as of this ____ day of __________________, 20____`; Section II Term begins `shall commence on ____________________`. No `August 1, 2023` appears in the viewed text. Assessment: Criterion requires removing any reference to the date August 1, 2023; pass because the date fields are blank/template fields and the specific date is absent.

  4. States all of the following: (1) that the target resident-to-staff ratio is 12:1 and (2) that temporary changes in the staff-to-resident ratio of up to 15:1 is permitted

    Pass

    Evidence: Section IV.J Staffing states: `including a target resident-to-staff ratio of approximately 12:1 with temporary variances up to 15:1 during low census or staffing shortages`. Assessment: Criterion requires stating both the target ratio is 12:1 and temporary changes up to 15:1 are permitted; pass because both values and the temporary variance permission are stated.

  5. States all of the following: (1) that individualized care plan assessments are performed every six months and (2) that individualized care plan assessments are performed as staffing allows

    Pass

    Evidence: Section IV.J states Community shall `provide an individualized care plan upon admission, conduct reassessments every six (6) months or as staffing allows`. Assessment: Criterion requires stating individualized care plan assessments are performed every six months and as staffing allows; pass because reassessments of the individualized care plan are stated as every six months or as staffing allows.

  6. States all of the following individuals may count to the overall coverage ratios if assisting with non-medical support tasks in the revised residency agreement: (i) volunteers at the residency, (ii) interns or other such individuals at the residency through an internship, apprenticeship or other such program, and (iii) family visitors of a resident

    Fail

    Evidence: Section IV.J only mentions staffing ratios and reassessments: `target resident-to-staff ratio... temporary variances... individualized care plan... reassessments... notify Resident's family...`. The viewed agreement contains no language that volunteers, interns/apprentices/program participants, or family visitors may count toward coverage ratios. Assessment: Criterion requires all three categories may count toward overall coverage ratios if assisting with non-medical support tasks; fail because this language is missing from the revised agreement.

  7. States that residents categorized as “Stable” may defer reassessment of their individualized care plan for up to twelve (12) months with administrative approval

    Fail

    Evidence: Section IV.J states reassessments occur `every six (6) months or as staffing allows`, but contains no statement about residents categorized as `Stable` deferring reassessment for up to twelve months with administrative approval. Assessment: Criterion requires the Stable-category twelve-month deferral language; fail because it is not present in the artifact.