APEX-Agents · Law
World415_im_01
APEX-Agents task World415_im_01 in AI Agents for SEC Disclosure Analysis. Compare dual-harness agent runs across models — rubric criteria, scores, and public traces.
Task prompt
What the agent was asked to do
An independent investigation of Crowdstrike's Channel File 291 outage revealed that the devices affected were mostly Microsoft or Google devices, and no Apple devices were affected at all. Is another filing with the SEC required at this time? Please give me a yes/no with a clear explanation back here so I can understand your answer and what the legal basis is for your assessment.
Published trajectories
Agent runs on this task
Curated dual-harness runs (parsed + original sandbox). Best scored run per model.
| Model | Harness | Score | Result | Links |
|---|---|---|---|---|
| GPT-5.5showcase | dual | 1/6 | Fail | Share pagePublic trace |
| fireworks models Kimi K2 | dual | 1/6 | Fail | Share pagePublic trace |
| Gemini 3.1 Pro | dual | 1/6 | Fail | Share pagePublic trace |
| GPT-5.4 | dual | 2/6 | Fail | Share pagePublic trace |
| GPT-5.4 mini | dual | 1/6 | Fail | Share pagePublic trace |
| GPT-5.4 nano | dual | 1/6 | Fail | Share pagePublic trace |
Grading rubric
Criteria and grader verdict (showcase run)
States that Yes, CrowdStrike must file a new Form 8-K with the SEC
FailEvidence: TEXT_RESPONSE begins, “No — based on the facts stated, another SEC filing is not required at this time,” and bottom line says “No immediate additional SEC filing is required.” Assessment: Criterion requires stating “Yes, CrowdStrike must file a new Form 8-K with the SEC”; fail because the response states the opposite.
States that CrowdStrike's current disclosures only disclose the impact of the Channel File 291 outage on Windows system devices
PassEvidence: Response says CrowdStrike’s Form 8-K disclosed outages for customers “utilizing certain Windows systems” and “Certain Windows systems… were affected.” It also frames a potential issue as whether “earlier statements limited to ‘certain Windows systems’ were materially incomplete.” Assessment: Criterion requires stating current disclosures only disclose impact on Windows system devices; pass, the response clearly conveys that the SEC filing/current disclosure was Windows-focused/limited.
States that a major portion of the devices affected by the Channel File 291 outage were Google devices
FailEvidence: The response references the user’s fact as “affected devices were ‘mostly Microsoft or Google devices’” and later says “if the ‘Google devices’ finding means…” Assessment: Criterion requires stating that a major portion of affected devices were Google devices. The response only states the combined category “mostly Microsoft or Google,” not that Google devices themselves were a major portion. Fail.
States that the impact of the Channel File 291 outage on Google devices is material information for CrowdStrike's investors
FailEvidence: Response states “the finding that affected devices were ‘mostly Microsoft or Google devices’ also does not, by itself, appear to change the already-disclosed material impact” and calls it “a non-material clarification.” It says reassess only “if ‘Google devices’ reflects a materially broader affected population.” Assessment: Criterion requires stating Google-device impact is material information for investors; fail because the response says it is not material on stated facts.
States that omitting material information from the Company's disclosures to the SEC violates the Exchange Act
FailEvidence: Response says SEC rules require more information “when needed to keep required statements from being misleading” and quotes that “further material information… [may be] necessary to make the required statements… not misleading.” It does not state that omitting material information from SEC disclosures violates the Exchange Act. Assessment: Criterion requires that specific legal conclusion; fail due to omission.
States both of the following regarding the new Form 8-K: (1) that it is a corrective filing pursuant to Item 8.01 and (2) that it clearly labels that this new Form 8-K is a correction to CrowdStrike's Form 8-K dated July 19, 2024
FailEvidence: Response states no new filing is required and discusses only Form 8-K Item 1.05 and a possible “new filing” if material facts emerge. It does not mention Item 8.01 or labeling a new Form 8-K as a correction to the July 19, 2024 Form 8-K. Assessment: Criterion requires both corrective Item 8.01 filing and clear correction label; fail.