Raycaster / evalsBack to AI Agents for Privacy and GDPR Compliance

APEX-Agents · Law

Law_World_423_DM_04

7/10Fail

APEX-Agents task Law_World_423_DM_04 in AI Agents for Privacy and GDPR Compliance. Compare dual-harness agent runs across models — rubric criteria, scores, and public traces.

AI Agents for Privacy and GDPR ComplianceLaw World 423Dual harnessGrader: rubric
task_3e4eba49c9cd4d3fa0de912ab1b1501e
Law World 423
message_in_console
7 models · dual config

Task prompt

What the agent was asked to do

Northstar's CEO sent me an email asking for a summary of the company’s liability under US privacy law if the incident occurred to a US customer based in Colorado. Please take the lead on drafting a high-level follow-up email to our CEO. Reply to me with it here and I'll review. In your draft email, identify the relevant sections of the Colorado Privacy Act that may have been violated and any underlying facts supporting each determination.

Published trajectories

Agent runs on this task

Curated dual-harness runs (parsed + original sandbox). Best scored run per model.

ModelHarnessScoreResultLinks
GPT-5.5showcasedual7/10Fail
fireworks models Kimi K2dual8/10Fail
Gemini 3 Flashdual7/10Fail
Gemini 3.1 Produal8/10Fail
GPT-5.4dual2/10Fail
GPT-5.4 minidual8/10Fail
GPT-5.4 nanodual0/10Fail

Grading rubric

Criteria and grader verdict (showcase run)

  1. States Yes, the incident would likely result in violations under Colorado privacy law

    Fail

    Evidence: The draft says “CPA exposure is possible” and emphasizes “the threshold issue is important,” then concludes only “if CPA applies” regulators may argue duties were failed. Assessment: The criterion requires stating “Yes, the incident would likely result in violations under Colorado privacy law.” The response is conditional and does not clearly state that violations would likely result. Fail.

  2. States that Northstar breached its duty of transparency under the Colorado Privacy Act § 6-1-1308(1))

    Pass

    Evidence: The draft identifies “C.R.S. § 6-1-1308(1) — Duty of transparency” and says there is “a plausible transparency issue,” with final exposure including “failed the CPA’s duty of care, transparency...” Assessment: The criterion asks whether it states Northstar breached the transparency duty. The response clearly flags this section as a likely/possible violated duty with supporting facts. Pass.

  3. States that Northstar breached its duty of purpose specification under the Colorado Privacy Act § 6-1-1308(2)

    Pass

    Evidence: The draft lists “C.R.S. § 6-1-1308(2)–(4) — Purpose specification, data minimization, and secondary use” and says the production transmission could be “an unspecified or incompatible secondary use.” Assessment: The criterion requires stating breach of purpose specification under § 6-1-1308(2). The response identifies that duty and applies facts suggesting unspecified processing. Pass.

  4. States that Northstar's use of data was beyond the scope of the consent provided for Northstar's collection of the data

    Pass

    Evidence: The draft states the analytics processing may be improper “unless it was reasonably necessary to the disclosed service purpose or supported by valid consumer consent,” and notes “there is no indication that consumers consented to this secondary analytics processing.” Assessment: The criterion asks whether it states the use was beyond the scope of consent. This is clearly conveyed. Pass.

  5. States that Northstar breached its duty of data minimization under the Colorado Privacy Act § 6-1-1308(3)

    Pass

    Evidence: The draft expressly includes “data minimization” in “C.R.S. § 6-1-1308(2)–(4)” and supports it with event-level metadata tied to “user UUIDs and device IDs” sent for a dev/staging module active in production. Assessment: The criterion requires stating breach of data minimization under § 6-1-1308(3). The response identifies that duty as part of the likely exposure and provides supporting facts. Pass.

  6. States that Northstar breached its duty to avoid secondary use under the Colorado Privacy Act § 6-1-1308(4)

    Pass

    Evidence: The draft labels “secondary use” under “C.R.S. § 6-1-1308(2)–(4)” and states a regulator could view the transmission as “an unspecified or incompatible secondary use.” Assessment: The criterion asks whether it states breach of the duty to avoid secondary use under § 6-1-1308(4). The response clearly does so. Pass.

  7. States that Northstar's security measures failed to adequately safeguard the data that Northstar collected

    Pass

    Evidence: The draft states the strongest issue is security/duty of care, citing that “the debug flag was not environment-scoped,” “outbound traffic was not monitored or alerted,” and “diagnostic logs were not ingested into SIEM.” Assessment: The criterion asks whether it states security measures failed to adequately safeguard collected data. This is directly supported. Pass.

  8. States that Northstar breached its duty of care under the Colorado Privacy Act § 6-1-1308(5)

    Pass

    Evidence: The draft lists “C.R.S. § 6-1-1308(5) — Duty of care” and says “This is likely the strongest CPA issue,” with facts about controls and monitoring failures. Assessment: The criterion requires stating breach of duty of care under § 6-1-1308(5). The response clearly identifies this as a likely violation/exposure. Pass.

  9. States that Northstar may not have obtained the requisite consent to process a consumer's sensitive data

    Fail

    Evidence: The draft says sensitive-data consent “is not currently a likely violation” because transmitted fields did not include “health data, medical information... or other sensitive data,” adding it only “becomes relevant if” different Colorado facts included sensitive data. Assessment: The criterion requires stating Northstar may not have obtained requisite consent to process sensitive data. The response does not state that for the incident; it largely negates it on current facts. Fail.

  10. States that Northstar may have breached its duty regarding sensitive data under the Colorado Privacy Act § 6-1-1308(7)

    Fail

    Evidence: The draft identifies “C.R.S. § 6-1-1308(7) — Sensitive data consent,” but says “this is not currently a likely violation” and only conditional if the Colorado version included sensitive data. Assessment: The criterion asks whether it states Northstar may have breached the sensitive-data duty under § 6-1-1308(7). The response does not state this as a possible breach on the documented incident facts. Fail.