Raycaster / evalsBack to AI Agents for SEC Disclosure Analysis

APEX-Agents · Law

Task awys8050

6/8Fail

APEX-Agents task Task awys8050 in AI Agents for SEC Disclosure Analysis. Compare dual-harness agent runs across models — rubric criteria, scores, and public traces.

AI Agents for SEC Disclosure AnalysisLaw World 415Dual harnessGrader: rubric
task_0cf6902f9ef341328e3261f7055918ee
Law World 415
make_new_doc
7 models · dual config

Task prompt

What the agent was asked to do

CrowdStrike's general counsel sent us a complaint filed in U.S. district court by Larry Stone, alleging violations under Sections 10(b) and 20(a) of the Securities Exchange Act of 1934, as well as Rule 10b-5 arising from false statements or omissions regarding its Falcon Sensor, the update of which causing the widely-reported July, 2024 service outage, leading to his Class A stock suffering a considerable loss in value. Review our directories and the attached file for analysis and reply back to me with a short memo in a new dox file. Determine whether the Plymouth matter's class, which is pending certification and does not show a related opt out, is likely to support a successful motion to dismiss Stone's suit.

Published trajectories

Agent runs on this task

Curated dual-harness runs (parsed + original sandbox). Best scored run per model.

ModelHarnessScoreResultLinks
GPT-5.5showcasedual6/8Fail
fireworks models Kimi K2dual5/8Fail
Gemini 3 Flashdual8/8Pass
Gemini 3.1 Produal3/8Fail
GPT-5.4dual0/8Fail
GPT-5.4 minidual1/8Fail
GPT-5.4 nanodual7/8Fail

Grading rubric

Criteria and grader verdict (showcase run)

  1. States that Larry purchased 56 shares

    Pass

    Evidence: In `/root/workspace/filesystem/Stone_Plymouth_Dismissal_Memo.docx`, Key facts reviewed states: “Larry Stone as purchasing 56 CrowdStrike Class A shares on April 3, 2024.” Assessment: The criterion asks whether it states Larry purchased 56 shares; pass because the memo explicitly states 56 shares.

  2. States that Larry purchased Class A shares

    Pass

    Evidence: `/root/workspace/filesystem/Stone_Plymouth_Dismissal_Memo.docx`, Key facts reviewed: “Larry Stone as purchasing 56 CrowdStrike Class A shares on April 3, 2024.” Assessment: The criterion asks whether it states Larry purchased Class A shares; pass because the memo explicitly identifies the shares as CrowdStrike Class A shares.

  3. States that Larry purchased shares on April 3, 2024

    Pass

    Evidence: `/root/workspace/filesystem/Stone_Plymouth_Dismissal_Memo.docx`, Key facts reviewed: “Larry Stone as purchasing 56 CrowdStrike Class A shares on April 3, 2024.” Assessment: The criterion asks whether it states Larry purchased shares on April 3, 2024; pass because the memo gives that exact date.

  4. States that class action member plaintiffs are those who purchased Class A shares of CrowdStrike stock between November 29, 2023 and July 29, 2024

    Pass

    Evidence: `/root/workspace/filesystem/Stone_Plymouth_Dismissal_Memo.docx`, Key facts reviewed: “The original Plymouth complaint asserted... claims for purchasers of CrowdStrike Class A common stock from November 29, 2023 through July 29, 2024.” Assessment: The criterion asks whether it states class action member plaintiffs are those who purchased Class A shares between November 29, 2023 and July 29, 2024; pass because the original Plymouth class scope is stated with those securities and dates.

  5. States that Larry is a member of the class

    Pass

    Evidence: `/root/workspace/filesystem/Stone_Plymouth_Dismissal_Memo.docx`, Key facts reviewed states Stone’s purchase “falls within both the original Plymouth period and the consolidated class period,” and Bottom line refers to Stone being “likely within the pending Plymouth/In re CrowdStrike putative class.” Assessment: The criterion asks whether it states Larry is a member of the class; pass because the memo conveys Stone falls within/is likely within the putative class.

  6. States that Larry did not opt out of the class

    Pass

    Evidence: `/root/workspace/filesystem/Stone_Plymouth_Dismissal_Memo.docx`, Key facts reviewed: “No class-certification order, Rule 23(b)(3) class notice... or Stone opt-out record was located”; Bottom line says “the file does not show a Stone opt-out opportunity.” Assessment: The criterion asks whether it states Larry did not opt out of the class; pass because the memo states no Stone opt-out record/opportunity was found.

  7. States that the suit can be dismissed

    Fail

    Evidence: `/root/workspace/filesystem/Stone_Plymouth_Dismissal_Memo.docx`, Bottom line: “A motion to dismiss Stone solely because he is likely within the pending... class is unlikely to succeed”; Conclusion: “not likely by itself to support a successful motion to dismiss Stone’s suit.” Assessment: The criterion asks whether it states the suit can be dismissed; fail because the memo states the opposite as to dismissal based on Plymouth/class membership, recommending transfer/consolidation/stay instead.

  8. States that the grounds for dismissal are under the federal claim-splitting/duplicative-litigation doctrine

    Fail

    Evidence: `/root/workspace/filesystem/Stone_Plymouth_Dismissal_Memo.docx`, Legal analysis section heading says “Preclusion and claim-splitting arguments are premature,” and explains likely remedies are “a stay, transfer, consolidation, or management order—not Rule 12 dismissal with prejudice.” Assessment: The criterion asks whether it states the grounds for dismissal are under the federal claim-splitting/duplicative-litigation doctrine; fail because the memo does not assert that doctrine as grounds for dismissal and instead says claim-splitting/duplicative-litigation dismissal is premature/unlikely.