Raycaster / evalsBack to AI Agents for Tax Due Diligence

APEX-Agents · Law

World425_tas_07

6/9Fail

APEX-Agents task World425_tas_07 in AI Agents for Tax Due Diligence. Compare dual-harness agent runs across models — rubric criteria, scores, and public traces.

AI Agents for Tax Due DiligenceLaw World 425Dual harnessGrader: rubric
task_f8f47a9c94874854a24936d81a89fdfb
Law World 425
make_new_doc
6 models · dual config

Task prompt

What the agent was asked to do

We have been asked to determine whether the S-corporation election of Summit Filing Solutions, Inc. ("Summit"), which Summit claims was effective on 1/1/19, was terminated. Please analyze Summit's Shareholder Agreement, Summit's Income Schedule, and the Office Lease Agreement between Summit and Anderson Instruments. Please disregard issues arising under 26 U.S.C. sec. 1361(b)(1). Please prepare a concise memorandum in a new DOCX file you make, briefly explaining your conclusions and citing to appropriate authority.

Published trajectories

Agent runs on this task

Curated dual-harness runs (parsed + original sandbox). Best scored run per model.

ModelHarnessScoreResultLinks
GPT-5.5showcasedual6/9Fail
fireworks models Kimi K2dual3/9Fail
Gemini 3.1 Produal4/9Fail
GPT-5.4dual6/9Fail
GPT-5.4 minidual3/9Fail
GPT-5.4 nanodual4/9Fail

Grading rubric

Criteria and grader verdict (showcase run)

  1. States that Summit's S-corporation election was terminated

    Pass

    Evidence: DOCX brief answer says, “Subject to the accumulated-earnings-and-profits caveat below, Summit's S election terminated effective January 1, 2022,” and conclusion repeats it terminated “if Summit had accumulated C corporation earnings and profits.” Assessment: Criterion asks whether it states Summit's S-corporation election was terminated. Pass, though the memo states the conclusion conditionally.

  2. States that under 26 U.S.C. § 1362(d)(3), where a corporation has retained earnings and profits derived during a period preceding its S-corporation election for each of three consecutive taxable years and has, during each of those three years, passive investment income which exceeds 25% of its gross receipts, its S-corporation election is terminated

    Pass

    Evidence: Governing law section states termination applies when “for each of three consecutive taxable years, the corporation (i) has accumulated earnings and profits at year-end and (ii) has gross receipts more than 25 percent of which are passive investment income,” citing I.R.C. § 1362(d)(3)(A), and that it is effective after the third year. Assessment: Criterion asks for this rule under 26 U.S.C. § 1362(d)(3). Pass; the memo clearly states the three-year accumulated E&P/passive income rule, despite using “accumulated earnings and profits” rather than the rubric’s “retained” wording.

  3. States that under 26 C.F.R. § 1.1362-2(c)(5), passive investment income includes gross receipts derived from property rental

    Pass

    Evidence: Governing law section says, “Passive investment income generally includes gross receipts derived from rents. I.R.C. section 1362(d)(3)(C)(i); Treas. Reg. section 1.1362-2(c)(5)(i).” Assessment: Criterion asks whether it states that passive investment income includes gross receipts derived from property rental under 26 C.F.R. § 1.1362-2(c)(5). Pass.

  4. States that although 26 C.F.R. § 1.1362-2(c)(5)(ii)(B)(2) provides that rents derived in the active trade or business of renting property are generally excepted from passive income, the exception applies to circumstances in which the corporation provides significant services or incurs substantial costs in its rental activities and excludes “net” leases

    Pass

    Evidence: The memo states the rental exception “requires significant services or substantial costs in the rental business,” and that regulations provide such services/costs generally are not present “in connection with net leases.” Assessment: Criterion asks for the active rental exception, significant services/substantial costs, and exclusion of net leases. Pass.

  5. States that Summit’s Gross Income Schedule shows that Summit had earnings and profits from tax year 2018, which preceded its S-Corporation election

    Fail

    Evidence: Application section says, “The reviewed documents establish a pre-S C corporation period in 2018 and show substantial 2018 gross receipts, but they do not by themselves establish Summit's accumulated earnings and profits...” Assessment: Criterion asks whether it states Summit’s Gross Income Schedule shows earnings and profits from tax year 2018. Fail; the memo expressly distinguishes gross receipts from E&P and says the schedule does not establish E&P.

  6. States that Summit did all of the following which retained Summit's 2018 earnings and profits: minimize expenses, paying no payroll to shareholder-employees until 2024, and making no distributions to shareholders until early 2025

    Fail

    Evidence: The DOCX contains no statement that Summit minimized expenses, paid no payroll to shareholder-employees until 2024, or made no distributions until early 2025; instead it says E&P requires confirmation from “Form 1120, E&P workpapers, distribution records, or other corporate-tax records.” Assessment: Criterion asks for all listed actions retaining 2018 E&P. Fail.

  7. States that Summit would have retained its 2018 earnings and profits for at least three years after its S-Corporation election

    Fail

    Evidence: The memo says “Accumulated E&P is the remaining factual condition” and that reviewed documents “do not by themselves establish Summit's accumulated earnings and profits at the close of 2019, 2020, and 2021.” Assessment: Criterion asks whether it states Summit would have retained 2018 E&P for at least three years after the S election. Fail; the memo states this only as an unconfirmed condition, not a conclusion.

  8. States that Summit’s gross receipts derived from property rental exceeded 25% of Summit’s gross receipts for each of the three years following 2018

    Pass

    Evidence: Application section table lists passive rent percentages for 2019, 2020, and 2021 of 32.4%, 32.0%, and 31.3%, and says “The base-rent percentage exceeded 25 percent in each of those years.” Assessment: Criterion asks whether it states property-rental gross receipts exceeded 25% for each of the three years following 2018. Pass.

  9. States that the triple-net provisions of the Office Lease Agreement lead to the conclusion that the exception of 26 C.F.R. § 1.1362-2(c)(5)(ii)(B)(2) does not apply to the rental income derived through that lease

    Pass

    Evidence: Brief answer states the lease is “a triple-net arrangement” and rent is passive unless services/costs exception applies; Application states additional-rent provisions make it “a net lease in substance” and “Summit is not providing significant services or incurring substantial unreimbursed costs,” so base rent is passive. Assessment: Criterion asks whether triple-net lease provisions lead to conclusion that the regulatory exception does not apply. Pass.