Raycaster / evalsBack to AI Agents for Tax Due Diligence

APEX-Agents · Law

World425_amk_01

6/6Pass

APEX-Agents task World425_amk_01 in AI Agents for Tax Due Diligence. Compare dual-harness agent runs across models — rubric criteria, scores, and public traces.

AI Agents for Tax Due DiligenceLaw World 425Dual harnessGrader: rubric
task_8d501efe0f924f69aeee070f2e08b576
Law World 425
message_in_console
7 models · dual config

Task prompt

What the agent was asked to do

Summit filed Form 2553 (Rev B) with the Internal Revenue Service ("IRS") on February 19, 2019. Do you see any potential problems under Treasury Regulation 1.1362-6(b)? Give me your analysis back here so I can decide what to do.

Published trajectories

Agent runs on this task

Curated dual-harness runs (parsed + original sandbox). Best scored run per model.

ModelHarnessScoreResultLinks
GPT-5.5showcasedual6/6Pass
fireworks models Kimi K2dual0/6Fail
Gemini 3 Flashdual6/6Pass
Gemini 3.1 Produal6/6Pass
GPT-5.4dual1/6Fail
GPT-5.4 minidual1/6Fail
GPT-5.4 nanodual1/6Fail

Grading rubric

Criteria and grader verdict (showcase run)

  1. States that Treasury Regulation 1.1362-6(b)(2)(i) requires that when the stock of the corporation is owned by husband and wife as community property, each spouse must consent to the election

    Pass

    Evidence: Response states, “Under § 1.1362-6(b)(2)(i), if Gregory was married and the stock or income from the stock was community property, his spouse also needed to consent,” and earlier quotes that “each person having a community interest... must consent.” Assessment: The criterion asks whether it states the regulation requires each spouse to consent when stock is owned by husband and wife as community property. This is clearly conveyed. Pass.

  2. States that Wisconsin is a community property state

    Pass

    Evidence: Response states, “Wisconsin is a community-property state for federal community-property purposes,” and notes the IRS publication list includes “Wisconsin.” Assessment: The criterion asks whether it states Wisconsin is a community property state. Pass.

  3. States that IRS Form 2553 provides that shareholder Gregory Beaumont resides in a community property state

    Pass

    Evidence: Response states, “Gregory Beaumont is listed at a Wisconsin address on the shareholder records and the Form 2553 materials” and then discusses Wisconsin as a community-property state. Assessment: The criterion asks whether it states IRS Form 2553 provides Gregory Beaumont resides in a community property state. The response specifically ties Gregory’s Wisconsin address to Form 2553 materials, satisfying this chat-coverage item. Pass.

  4. States that Gregory Beaumont's marital status is unclear as of February 19, 2019

    Pass

    Evidence: Response states, “This is contingent: if Gregory was unmarried... there may be no issue” and recommends verifying “marital status... as of the election date.” It also says “if Gregory was married...” rather than asserting he was. Assessment: The criterion asks whether it states Gregory Beaumont's marital status is unclear as of February 19, 2019. Although not using the word “unclear,” the response clearly conveys uncertainty and need to verify as of the election date. Pass.

  5. States that if Gregory Beaumont was married as of February 19, 2019, a consent from his spouse is potentially missing

    Pass

    Evidence: Response states, “if Gregory was married and the stock or income from the stock was community property, his spouse also needed to consent. I did not see a spouse consent on Rev B.” It summarizes “possible missing community-property spouse consent for Gregory Beaumont due to Wisconsin.” Assessment: The criterion asks whether it states that if Gregory was married as of February 19, 2019, spouse consent is potentially missing. This is clearly stated with the relevant contingency. Pass.

  6. States that Gregory Beaumont's marital status at the time of signing Form 2553, February 19, 2019 should be confirmed

    Pass

    Evidence: Response recommends verifying “marital status, domicile, and property characterization as of the election date” and in next steps: “Marital/community-property confirmations for each 2019 shareholder, especially Gregory Beaumont.” Assessment: The criterion asks whether it states Gregory’s marital status at the time of signing Form 2553, February 19, 2019 should be confirmed. The response says to verify his marital status as of the election date/2019 shareholder context. Pass.