Raycaster / evalsBack to AI Agents for Tax Due Diligence

APEX-Agents · Law

World425_jcf-03

6/7Fail

APEX-Agents task World425_jcf-03 in AI Agents for Tax Due Diligence. Compare dual-harness agent runs across models — rubric criteria, scores, and public traces.

AI Agents for Tax Due DiligenceLaw World 425Dual harnessGrader: rubric
task_b68a970f95ea48019176f0be1f73e61b
Law World 425
make_new_doc
7 models · dual config

Task prompt

What the agent was asked to do

Review Summit's records to determine how Summit can correct any potential questions about the validity of its S corporation tax election. Write a tax memo and put it in a New document (docx) for me to review later.

Published trajectories

Agent runs on this task

Curated dual-harness runs (parsed + original sandbox). Best scored run per model.

ModelHarnessScoreResultLinks
GPT-5.5showcasedual6/7Fail
fireworks models Kimi K2dual3/7Fail
Gemini 3 Flashdual7/7Pass
Gemini 3.1 Produal6/7Fail
GPT-5.4dual4/7Fail
GPT-5.4 minidual4/7Fail
GPT-5.4 nanodual5/7Fail

Grading rubric

Criteria and grader verdict (showcase run)

  1. States that Summit elected to be taxed as an S corporation

    Pass

    Evidence: In /root/workspace/filesystem/Memorandum/Summit_S_Corporation_Election_Correction_Tax_Memo.docx, Short Answer says “Summit should…preserve or substantiate S corporation status,” and Facts Reviewed states “Summit records show an intended S corporation election effective January 1, 2019.” Assessment: The criterion requires stating Summit elected to be taxed as an S corporation. Pass: the memo clearly states Summit had/intended an S election effective Jan. 1, 2019.

  2. States that the tax code prohibits an S corporation from having a nonresident alien as a shareholder,

    Pass

    Evidence: In the memo’s Applicable Law section: “A small business corporation…cannot have a nonresident alien shareholder.” Assessment: The criterion requires stating the tax code prohibits an S corporation from having a nonresident alien shareholder. Pass: the memo states the prohibition under IRC Section 1361(b)(1).

  3. States that a transfer of Summit shares to a nonresident alient would automatically terminate Summit's S corporation tax election

    Pass

    Evidence: Short Answer states: “If Ms. Canuck was a nonresident alien, Summit ceased to be a small business corporation on that date and its S election terminated absent relief.” Applicable Law states if an S corporation ceases to be a small business corporation, “the S election terminates effective on the date of cessation.” Assessment: The criterion requires stating a transfer of Summit shares to a nonresident alien would automatically terminate the S election. Pass: the memo makes this point clearly, though using “terminated absent relief” rather than “automatically.”

  4. States the transfer of Summit shares to Canuck was likely an inadvertent termination of S Corporation status

    Pass

    Evidence: Under “Resolve the Carrie Canuck transfer,” the memo calls it “the principal potential termination event” and recommends, if Canuck was not a U.S. citizen or resident, a PLR “describing the transfer as inadvertent, the transfer back as corrective action, and the absence of tax-avoidance intent.” Assessment: The criterion requires stating the transfer to Canuck was likely an inadvertent termination of S corporation status. Pass: the memo characterizes the transfer as a potential termination event and directs it be presented as inadvertent for Section 1362(f) relief.

  5. States that Canuck transferred her Summit shares back to Beaumont within less than a year

    Pass

    Evidence: Facts Reviewed states Beaumont transferred 100,000 shares to Canuck on “June 1, 2022” and “Ms. Canuck transferred the shares back to Mr. Beaumont on February 4, 2023.” Short Answer also notes the “February 4, 2023 transfer back.” Assessment: The criterion requires stating Canuck transferred her Summit shares back to Beaumont within less than a year. Pass: the memo gives dates showing a return in about eight months.

  6. States that prompt return of shares to Beaumont may be deemed to satisfy the requirement of prompt corrective action following discovery of an event causing inadvertent termination of a corporation's S Corporation status

    Pass

    Evidence: Applicable Law states Section 1362(f) relief requires that “within a reasonable period after discovery, steps were taken so the corporation is a small business corporation.” The recommended PLR should describe “the transfer back as corrective action,” and favorable facts include “the Canuck shares were transferred back.” Assessment: The criterion requires stating prompt return of shares to Beaumont may satisfy the prompt corrective action requirement after discovery. Pass: the memo links the return of shares to corrective action under the Section 1362(f) relief requirements, though it does not use the exact phrase “prompt corrective action.”

  7. States that the only available remedy is for Summit to seek an IRS ruling (private letter ruling) on the validity of its S corporation election

    Fail

    Evidence: The memo recommends multiple potential actions: “verify or replace the IRS acceptance evidence,” “file a corrected Form 2553 package under Rev. Proc. 2013-30 if eligible,” use “Rev. Proc. 2004-35 automatic relief” for some spouse consents, and pursue a Section 1362(f) PLR if Canuck was an NRA or Rev. B binding. Assessment: The criterion requires stating the only available remedy is for Summit to seek an IRS private letter ruling on the validity of its S election. Fail: the memo does not state PLR is the only available remedy; it presents several possible remedies and treats PLR as safer/conditional for certain issues.