APEX-Agents · Law
World419_UM_03
APEX-Agents task World419_UM_03 in AI Agents for Maritime and Environmental Liability. Compare dual-harness agent runs across models — rubric criteria, scores, and public traces.
Task prompt
What the agent was asked to do
Following the M/V Red Room incident, Cooper/Jeffries incurred $1,800,000 in cleanup costs. The company equally settled a claim in the amount of $295,000 from Captain Jack Sparrow, a fisherman whose boat and equipment were damaged as a result of the incident; Cooper/Jeffries paid a $128,375 penalty on behalf of Martin Dee, captain of the M/V Red Room, for his role in the incident. Cooper/Jeffries seeks to recoup all sums paid from Black Lodge Petroleum Logistics ("BLPL"). Review all the insurance policies contained in the Risk Management Counsel folder and BLPL's Umbrella Insurance Policy. Concisely determine which, if any, of the insurance policies will cover the cleanup costs incurred as a result of the M/V Red Room incident, and the amount paid to settle Sparrow's claim. Reply to me with your assessment.
Published trajectories
Agent runs on this task
Curated dual-harness runs (parsed + original sandbox). Best scored run per model.
| Model | Harness | Score | Result | Links |
|---|---|---|---|---|
| GPT-5.5showcase | dual | 4/6 | Fail | Share pagePublic trace |
| fireworks models Kimi K2 | dual | 4/6 | Fail | Share pagePublic trace |
| Gemini 3 Flash | dual | 6/6 | Pass | Share pagePublic trace |
| Gemini 3.1 Pro | dual | 6/6 | Pass | Share pagePublic trace |
| GPT-5.4 | dual | 6/6 | Pass | Share pagePublic trace |
| GPT-5.4 mini | dual | 6/6 | Pass | Share pagePublic trace |
| GPT-5.4 nano | dual | 6/6 | Pass | Share pagePublic trace |
Grading rubric
Criteria and grader verdict (showcase run)
States that the Protection & Indemnity Insurance Policy will cover the cleanup costs incurred by Cooper/Jeffries as a result of the M/V Red Room incident
FailEvidence: The response identifies “BLPL’s Terminal Operators Liability and Marine Pollution Legal Liability coverage” as the “best recovery target” and says Terminal Operators “covers both.” It does not mention any “Protection & Indemnity Insurance Policy.” Assessment: Criterion requires stating that the Protection & Indemnity policy will cover cleanup costs. Fail because the response does not make that statement.
States that the Marine Pollution Legal Liability Insurance Policy will cover the cleanup costs incurred by Cooper/Jeffries as a result of the M/V Red Room incident
PassEvidence: The response states “BLPL Marine Pollution Legal Liability Policy — covers both” and that it covers “Legal liability for pollution damage, cleanup costs, and related expenses arising from a pollution incident.” Assessment: Criterion requires stating Marine Pollution Legal Liability will cover cleanup costs. Pass.
States that the Protection & Indemnity Insurance Policy will cover the costs incurred by Cooper/Jeffries to settle Captain Jack Sparrow's claim
FailEvidence: For Sparrow’s claim, the response says “Terminal Operators Liability Policy — covers both” and “Marine Pollution Legal Liability Policy — covers both,” but does not mention a Protection & Indemnity policy. Assessment: Criterion requires stating that Protection & Indemnity will cover Sparrow settlement costs. Fail because that policy is absent from the assessment.
States that the Marine Pollution Legal Liability Insurance Policy will cover the costs incurred by Cooper/Jeffries to settle Captain Jack Sparrow's claim
PassEvidence: The response states “BLPL Marine Pollution Legal Liability Policy — covers both” and, “For Sparrow, it expressly covers pollution-caused third-party property damage,” including “Damage to vessels, boats, and marine equipment” and fishing gear. Assessment: Criterion requires stating Marine Pollution Legal Liability covers the Sparrow settlement. Pass.
States that the Marine Excess Liability Insurance Policy may provide excess coverage for the cleanup costs incurred by Cooper/Jeffries as a result of the M/V Red Room incident
PassEvidence: The response states “BLPL Marine Excess/Bumbershoot — only if underlying limits are exhausted” and “It may become available if the Terminal Operators / Marine Pollution layers are insufficient.” The overall assessment identifies cleanup costs as part of the covered recoupment target. Assessment: Criterion requires stating Marine Excess may provide excess coverage for cleanup costs. Pass.
States that the Marine Excess Liability Insurance Policy may provide excess coverage for the costs incurred by Cooper/Jeffries to settle Captain Jack Sparrow's claim
PassEvidence: The response states Marine Pollution covers Sparrow and then says “BLPL Marine Excess/Bumbershoot — only if underlying limits are exhausted” and “may become available if the Terminal Operators / Marine Pollution layers are insufficient.” Assessment: Criterion requires stating Marine Excess may provide excess coverage for the Sparrow settlement. In context, this excess statement applies to the underlying coverages for both covered losses, including Sparrow. Pass.