APEX-Agents · Law
LawWorld433_ANB_04
APEX-Agents task LawWorld433_ANB_04 in AI Agents for Healthcare and Senior Living Legal Risk. Compare dual-harness agent runs across models — rubric criteria, scores, and public traces.
Task prompt
What the agent was asked to do
Magnolia Gardens is looking to renew its 2025 coverage through Valiant Assurance Ltd. for 2026, in Texas. Valiant has asked for a list including the number of patients they serve that utilize Medicaid. Does Magnolia Gardens have to provide this information? State your view as a Yes or No, with a two-sentence explanation. Put it in a NEW documents document for me.
Published trajectories
Agent runs on this task
Curated dual-harness runs (parsed + original sandbox). Best scored run per model.
| Model | Harness | Score | Result | Links |
|---|---|---|---|---|
| GPT-5.5showcase | dual | 3/3 | Pass | Share pagePublic trace |
| fireworks models Kimi K2 | dual | 2/3 | Fail | Share pagePublic trace |
| Gemini 3.1 Pro | dual | 2/3 | Fail | Share pagePublic trace |
| GPT-5.4 | dual | 0/3 | Fail | Share pagePublic trace |
| GPT-5.4 mini | dual | 1/3 | Fail | Share pagePublic trace |
| GPT-5.4 nano | dual | 2/3 | Fail | Share pagePublic trace |
Grading rubric
Criteria and grader verdict (showcase run)
States that for purposes of writing professional liability insurance for healthcare providers, an insurance company may not consider the extent to which a health care provider provides services to individuals who are recipients of Medicaid
PassEvidence: In `/root/workspace/Magnolia Gardens Medicaid Disclosure View.docx`, the explanation states Texas Insurance Code Chapter 1901 "prohibits an insurer, when writing professional liability insurance, from considering whether or the extent to which the provider serves Medicaid recipients." Assessment: This satisfies the criterion that it states an insurance company may not consider the extent to which a health care provider provides services to Medicaid recipients for professional liability insurance.
States that for purposes of Title 10 §1901 of the Texas Insurance Code, an assisted living facility is considered a medical provider
PassEvidence: The document states: "Texas Insurance Code Chapter 1901 treats an assisted living facility as a health care provider." Assessment: Although it says "health care provider" rather than "medical provider," this directly conveys that an assisted living facility is within the provider category for Chapter 1901/Title 10 purposes, so the criterion is met.
States “No”, Magnolia Gardens does not have to provide documentation on its number of Medicaid patients to its insurer
PassEvidence: The document begins its answer with "No." and explains that because Valiant is requesting the Medicaid-utilization patient list/number for renewal, "Magnolia Gardens should not have to provide it for underwriting." Assessment: This meets the criterion to state “No” and that Magnolia Gardens does not have to provide documentation on its number of Medicaid patients to its insurer.