Raycaster / evalsBack to AI Agents for Tariff and Trade Law Analysis

APEX-Agents · Law

Task ymtecb81

8/9Fail

APEX-Agents task Task ymtecb81 in AI Agents for Tariff and Trade Law Analysis. Compare dual-harness agent runs across models — rubric criteria, scores, and public traces.

AI Agents for Tariff and Trade Law AnalysisLaw World 416Dual harnessGrader: rubric
task_bf18a6d6ff44489daf0b1c80b5d4a3a8
Law World 416
message_in_console
7 models · dual config

Task prompt

What the agent was asked to do

Can you take a look at the two Master Supply Agreement templates (Master Supply Agreement Template.pdf ("Template1"), Master Supply Agreement 2.pdf ("Template2"))? We’re considering them for Acme (the steel supplier) and we want a comparison. I need to know how each template deals with tariff‑related cost exposure, since Acme is importing steel from outside USMCA and the new tariffs are creating real financial pressure. Also, TAC is thinking about giving Acme a cash infusion secured by a lien on their receivables, but we’re worried about what happens if Acme goes bankrupt. Could you assess whether that financing structure would expose TAC to creditor claims, and which template gives TAC the most operational control? Please point to the clauses that support your analysis. Now, send me a clear text summary straight in here.

Published trajectories

Agent runs on this task

Curated dual-harness runs (parsed + original sandbox). Best scored run per model.

ModelHarnessScoreResultLinks
GPT-5.5showcasedual8/9Fail
fireworks models Kimi K2dual7/9Fail
Gemini 3 Flashdual4/9Fail
Gemini 3.1 Produal1/9Fail
GPT-5.4dual7/9Fail
GPT-5.4 minidual2/9Fail
GPT-5.4 nanodual5/9Fail

Grading rubric

Criteria and grader verdict (showcase run)

  1. States that Template1  is superior for insulating TAC from tariff exposure

    Pass

    Evidence: Response bottom line says “Template1 is much stronger for TAC on tariff exposure,” and tariff section says “Template1: tariff risk is effectively on Acme” with “TAC gets the strongest protection against tariff surcharges.” Assessment: The criterion asks whether it states Template1 is superior for insulating TAC from tariff exposure; pass because the response clearly states this.

  2. States that lien-secured financing would expose TAC to lender liability risk

    Pass

    Evidence: Response says the structure “creates bankruptcy and creditor-claim risk” and lists “Equitable subordination / lender-liability risk” if TAC uses the loan plus MSA rights to control Acme. Assessment: The criterion asks whether it states lien-secured financing would expose TAC to lender liability risk; pass because lender-liability risk is expressly identified.

  3. States that under Template1, TAC has absolute change rights to the Products provided by Acme

    Pass

    Evidence: Response lists Template1 operational control example: “Buyer change control: [Buyer may require changes to Products at any time for any reason].” Assessment: The criterion asks whether it states that under Template1 TAC has absolute change rights to Acme’s Products; pass because “at any time for any reason” conveys absolute buyer change rights.

  4. States that Template2 permits negotiated price adjustments for raw material cost changes

    Pass

    Evidence: Response says Template2 “allows raw-material cost adjustments to be negotiated” and quotes: “Price adjustments for raw material cost changes may be negotiated based on documented market indices and agreed-upon formulas, subject to Buyer's approval.” Assessment: The criterion asks whether it states Template2 permits negotiated price adjustments for raw material cost changes; pass.

  5. States that Template2 requires good faith negotiations over the cost and timing impacts of TAC‑initiated changes

    Pass

    Evidence: Response lists Template2 supplier protections: “Buyer-initiated changes require good-faith negotiation of cost/timing impacts” and quotes “Cost and timing impacts shall be negotiated in good faith.” Assessment: The criterion asks whether it states Template2 requires good faith negotiations over cost and timing impacts of TAC-initiated changes; pass.

  6. States that Template1 grants TAC unlimited rights to conduct financial audits of Acme

    Pass

    Evidence: Response says Template1 has “financial audit rights: [Buyer may audit Supplier's costs, pricing, and financial records at any time]” and “broad general audit rights: [Buyer has unlimited audit rights over Supplier].” Assessment: The criterion asks whether it states Template1 grants unlimited rights to conduct financial audits of Acme; pass because it states financial records can be audited at any time and also characterizes audit rights as unlimited.

  7. States that Template2 does not provide TAC with audit rights over Acme’s financial records

    Fail

    Evidence: The response says Template2 has “similar but less aggressive provisions” and mentions setoff, bankruptcy trigger, and independent contractor status, but does not state that Template2 lacks audit rights over Acme’s financial records. Assessment: The criterion asks whether it states Template2 does not provide TAC with audit rights over Acme’s financial records; fail because that specific point is absent.

  8. States that Template1 grants TAC greater operational control over Acme compared to Template2

    Pass

    Evidence: Response bottom line says “Template1 is much stronger for... operational control,” and Section 3 answers “Which template gives TAC the most operational control? Template1, by a wide margin.” Assessment: The criterion asks whether it states Template1 grants greater operational control than Template2; pass.

  9. States that with lien‑secured financing, Template2 is safer from a lender liability standpoint

    Pass

    Evidence: Recommendation says “If TAC’s priority is supporting Acme financially while reducing creditor-claim optics, Template2 is safer” and contrasts this with Template1’s “over-control optics.” Assessment: The criterion asks whether it states that with lien-secured financing, Template2 is safer from a lender liability standpoint; pass because it clearly states Template2 is safer for reducing creditor-claim/lender-liability optics in the financing context.