APEX-Agents · Law
LawWorld433_NAF_01
APEX-Agents task LawWorld433_NAF_01 in AI Agents for Healthcare and Senior Living Legal Risk. Compare dual-harness agent runs across models — rubric criteria, scores, and public traces.
Task prompt
What the agent was asked to do
Heather Collins has started to let her care staff all take naps at the same time. Legally, can she do this? Provide me with a brief explanation of your answer, citing the relevant sources. Reply back to me with your view in here.
Published trajectories
Agent runs on this task
Curated dual-harness runs (parsed + original sandbox). Best scored run per model.
| Model | Harness | Score | Result | Links |
|---|---|---|---|---|
| GPT-5.5showcase | dual | 2/5 | Fail | Share pagePublic trace |
| fireworks models Kimi K2 | dual | 1/5 | Fail | Share pagePublic trace |
| Gemini 3.1 Pro | dual | 3/5 | Fail | Share pagePublic trace |
| GPT-5.4 | dual | 3/5 | Fail | Share pagePublic trace |
| GPT-5.4 mini | dual | 1/5 | Fail | Share pagePublic trace |
| GPT-5.4 nano | dual | 1/5 | Fail | Share pagePublic trace |
Grading rubric
Criteria and grader verdict (showcase run)
States that Heather Collins cannot legally allow her care staff to all take naps at the same time
PassEvidence: TEXT_RESPONSE says, “My view: **No—not as described**” and “Letting **all** care staff nap at the same time would likely violate Texas assisted-living staffing rules.” Assessment: The criterion asks whether the response states Heather Collins cannot legally allow all care staff to nap at the same time; pass because the response clearly says no and explains simultaneous naps are not legally safe absent coverage.
States any of the following as the relevant law: (1) the Texas Administrative Code at section TAC 553.253(c)(4)(A); and (2) that in a large, Type A assisted living facility, night shift staff must be "immediately available and awake"
PassEvidence: TEXT_RESPONSE says, “in a large Type A facility, staff must be ‘immediately available and awake,’” and cites Texas assisted-living rules. Assessment: The criterion asks whether the response states relevant law as either TAC 553.253(c)(4)(A) or the rule that in a large Type A assisted living facility, night shift staff must be immediately available and awake. It states the latter substance, so pass.
States that Heather Collins manages the night shift at the Grove-Austin facility
FailEvidence: TEXT_RESPONSE refers to “Heather” generally but does not mention “the Grove-Austin facility” or that Heather Collins manages the night shift. Assessment: The criterion specifically requires stating that Heather Collins manages the night shift at Grove-Austin; fail because this fact is absent.
States that the Grove-Austin facility is licensed as a Type A assisted living facility under the Texas Administrative Code
FailEvidence: TEXT_RESPONSE discusses “a large Type A facility” and “Texas assisted-living staffing rules,” but never identifies “the Grove-Austin facility” or its license type. Assessment: The criterion requires stating Grove-Austin is licensed as a Type A assisted living facility under the Texas Administrative Code; fail because the response does not state this.
States that the Grove-Austin facility qualifies as a large assisted living facility under the Texas Administrative Code
FailEvidence: TEXT_RESPONSE says “in a large Type A facility” but does not state that Grove-Austin itself is large or qualifies as a large assisted living facility. Assessment: The criterion requires stating that the Grove-Austin facility qualifies as large under the Texas Administrative Code; fail because that facility-specific fact is missing.