Raycaster / evalsBack to AI Agents for Employment Law Analysis

APEX-Agents · Law

World425_jcf_02

5/10Fail

APEX-Agents task World425_jcf_02 in AI Agents for Employment Law Analysis. Compare dual-harness agent runs across models — rubric criteria, scores, and public traces.

AI Agents for Employment Law AnalysisLaw World 425Dual harnessGrader: rubric
task_0dab65657186412983c2c04e363fe40e
Law World 425
message_in_console
7 models · dual config

Task prompt

What the agent was asked to do

Review the shareholders agreement and identify any provisions that may cause issues with Summit's S-corp status. Identify corrective procedures that may be available to Summit. Print back what you find here.

Published trajectories

Agent runs on this task

Curated dual-harness runs (parsed + original sandbox). Best scored run per model.

ModelHarnessScoreResultLinks
GPT-5.5showcasedual5/10Fail
fireworks models Kimi K2dual3/10Fail
Gemini 3 Flashdual7/10Fail
Gemini 3.1 Produal5/10Fail
GPT-5.4dual3/10Fail
GPT-5.4 minidual6/10Fail
GPT-5.4 nanodual4/10Fail

Grading rubric

Criteria and grader verdict (showcase run)

  1. States that S corporations are permitted only one class of stock with identical governing provisions applying to all shares

    Pass

    Evidence: Response says an S corporation “cannot have more than one class of stock” and the test requires “identical rights to distribution and liquidation proceeds,” looking to “governing documents.” Assessment: Criterion requires stating S corps are permitted only one class with identical governing provisions applying to all shares. Pass; this is clearly conveyed.

  2. States that section 2.3 of the Shareholders Agreement results in a second class of stock through non‑identical governing provisions

    Pass

    Evidence: Response identifies “Highest-risk provision: special distributions to Michael Hartwell” and says “Because Section 2.3 is in the shareholder agreement and expressly relates to distributions, it likely creates a non-identical governing provision” that “can invalidate or terminate an S election.” Bottom line: “Section 2.3’s special distribution right… can create a second class of stock.” Assessment: Criterion requires stating section 2.3 results in a second class through non-identical governing provisions. Pass, despite qualified wording (“likely/can”), the position is clearly stated.

  3. States that section 2.3 of the Shareholder Agreement automatically terminates Summit's S corporation tax election

    Fail

    Evidence: Response says Section 2.3 “likely creates” a non-identical governing provision and “can invalidate or terminate an S election,” and bottom line says it “can create a second class of stock and jeopardize S-corp status.” Assessment: Criterion requires stating Section 2.3 automatically terminates Summit’s S corporation tax election. Fail; the response uses qualified risk language and does not state automatic termination.

  4. States that IRC Section 1362(f) allows for the IRS to create a process to provide relief from the inadvertent termination of a corporation’s S corporation tax election

    Pass

    Evidence: Response section C states “Summit can seek relief under IRC §1362(f)” and describes relief where “the IRS determines the circumstances were inadvertent” and corrective steps/adjustments occur. Assessment: Criterion requires stating IRC §1362(f) allows IRS to provide/process relief from inadvertent termination. Pass; the response identifies §1362(f) inadvertent termination relief and IRS role.

  5. States that Summit may be able to obtain relief for the inadvertent termination of its S corporation election under Rev. Proc. 2022-19

    Pass

    Evidence: Response states “Rev. Proc. 2022-19 provides a simplified retroactive correction procedure for certain non-identical governing provisions,” and says “this simplified path may not be available unless…” Assessment: Criterion requires stating Summit may be able to obtain relief under Rev. Proc. 2022-19. Pass; it presents the procedure as potentially available but likely limited based on facts.

  6. States that a corporation is eligible for corrective relief for the inadvertent termination of its S corporation election if it satisfies all of the following: the corporation has or had one or more non-identical governing provisions, the corporation has not made any disproportionate distributions, the corporation has timely filed a return on Form 1120-S for each applicable tax year, and the the non‑identical governing provision is corrected before it is discovered by the IRS

    Fail

    Evidence: Response notes eligibility requires no disproportionate distribution and discusses non-identical governing provisions; it also says amend “immediately” and quotes treatment if requirements met. It does not mention “timely filed a return on Form 1120-S for each applicable tax year” or that correction must occur “before it is discovered by the IRS.” Assessment: Criterion requires all four listed eligibility conditions. Fail because two required conditions are omitted.

  7. States that a corporation eligible for relief under Rev. Proc. 2022-19 may restore its S corporation tax election by completing all of the following: removing all non-identical governing provision before any non-identical governing provision is discovered by the IRS, completing the Corporate Governing Provision Statement described in Rev. Proc. 2022-19, and obtaining Shareholder Statements from all shareholders as described in Rev. Proc. 2022-19

    Fail

    Evidence: Response recommends amending to “Delete the Michael Hartwell special-distribution carveout,” but does not mention completing the “Corporate Governing Provision Statement” or obtaining “Shareholder Statements from all shareholders.” Assessment: Criterion requires all three procedural steps for Rev. Proc. 2022-19 relief. Fail; only removal/correction is covered.

  8. States that IRS will not issue a ruling for retroactive corrective relief regarding non-identical governing provisions if the corporation is eligible for relief under Rev. Proc. 2022-19

    Fail

    Evidence: Response says “A private letter ruling may be needed if Rev. Proc. 2022-19 is unavailable.” It does not state the IRS will not issue a ruling if the corporation is eligible for Rev. Proc. 2022-19 relief. Assessment: Criterion requires that specific no-ruling statement. Fail.

  9. States that if a corporation is ineligible for relief under Rev. Proc. 2022-19, it may seek relief under the Private Letter Ruling (“PLR”) process

    Pass

    Evidence: Response states “A private letter ruling may be needed if Rev. Proc. 2022-19 is unavailable” and bottom line says disproportionate distribution “may foreclose the streamlined Rev. Proc. 2022-19 correction and push Summit toward §1362(f) private-letter-ruling relief.” Assessment: Criterion requires stating that if ineligible for Rev. Proc. 2022-19, it may seek PLR relief. Pass.

  10. States that, in its request for a PLR, Summit must explain why it is ineligible for relief under Rev. Proc. 2022-19

    Fail

    Evidence: Response discusses why Rev. Proc. 2022-19 “may not be available” and that a PLR may be needed, but it does not state that Summit’s PLR request must explain why it is ineligible for Rev. Proc. 2022-19. Assessment: Criterion requires this specific PLR request content. Fail.