Raycaster / evalsBack to AI Agents for Cross-Border Regulatory Review

APEX-Agents · Law

World421_mc_08

9/10Fail

APEX-Agents task World421_mc_08 in AI Agents for Cross-Border Regulatory Review. Compare dual-harness agent runs across models — rubric criteria, scores, and public traces.

AI Agents for Cross-Border Regulatory ReviewLaw World 421Dual harnessGrader: rubric
task_3f9c85928e99498f818befbbdb35907c
Law World 421
make_new_doc
7 models · dual config

Task prompt

What the agent was asked to do

Can you create a GLBA-compliant privacy framework (and put it in a CONCISE documents document that you create new) for SLL's text messaging program that addresses initial privacy notice delivery via SMS, opt-out rights for non-public personal information sharing, annual notice requirements, and simplified notice eligibility?

Published trajectories

Agent runs on this task

Curated dual-harness runs (parsed + original sandbox). Best scored run per model.

ModelHarnessScoreResultLinks
GPT-5.5showcasedual9/10Fail
fireworks models Kimi K2dual6/10Fail
Gemini 3 Flashdual8/10Fail
Gemini 3.1 Produal6/10Fail
GPT-5.4dual7/10Fail
GPT-5.4 minidual7/10Fail
GPT-5.4 nanodual9/10Fail

Grading rubric

Criteria and grader verdict (showcase run)

  1. States that the initial consumer privacy notice must be provided not later than when Senior Living Lending, Inc. establishes a customer relationship

    Pass

    Evidence: /root/workspace/SLL_GLBA_SMS_Privacy_Framework.docx, section 4: “Timing. For customers, provide the initial notice no later than the time the customer relationship is established...” Assessment: The criterion asks whether it states the initial consumer privacy notice must be provided not later than when SLL establishes a customer relationship. Pass; the document states this timing rule for customers.

  2. States that SLL should use a hybrid delivery approach that includes all of the following: (1) SMS notification and (2) a URL link to the full privacy notice on a webpage

    Pass

    Evidence: /root/workspace/SLL_GLBA_SMS_Privacy_Framework.docx, bottom line: “Use SMS as a delivery and acknowledgment channel, not as the full GLBA privacy notice”; section 4: “The link must take the recipient directly to a printable/downloadable, version-controlled privacy notice”; sample SMS includes “SLL Privacy Notice: sll.com/privacy.” Assessment: The criterion requires a hybrid approach including SMS notification and URL link to the full notice webpage. Pass.

  3. States at least one of the following regarding SLL’s full privacy notice: (1) contains the complete Model Privacy Form and (2) complies with 12 C.F.R. Part 1016

    Pass

    Evidence: /root/workspace/SLL_GLBA_SMS_Privacy_Framework.docx, section 2: “Use the current GLBA model privacy notice, or counsel-approved equivalent, as the controlling notice”; source basis: “Regulation P, 12 CFR Part 1016...” Assessment: The criterion requires stating the full notice contains the complete Model Privacy Form or complies with 12 C.F.R. Part 1016. Pass; it directs use of the GLBA model privacy notice.

  4. States that SLL should obtain consumer consent for electronic delivery prior to delivering the privacy notices

    Pass

    Evidence: /root/workspace/SLL_GLBA_SMS_Privacy_Framework.docx, section 4: “Send the GLBA notice link by SMS only after the consumer has consented to electronic notices...” and section 2: “Electronic delivery through SMS requires the consumer/customer to agree to electronic delivery...” Assessment: The criterion requires consumer consent for electronic delivery before notices. Pass.

  5. States that SLL must provide consumers with a reasonable means to exercise an opt-out right

    Pass

    Evidence: /root/workspace/SLL_GLBA_SMS_Privacy_Framework.docx, section 5: “SLL must provide an initial privacy notice, a clear opt-out notice, a reasonable means to opt out...” and “Reasonable means. Offer at least two practical channels...” Assessment: The criterion requires a reasonable means to exercise opt-out. Pass.

  6. States at least two of the following mechanisms for opting out of non-public personal information sharing: (1) Designate check-off boxes in a prominent position on the relevant forms with the opt out notice; (2) Include a reply form together with the opt out notice that includes the address to which the form should be mailed; (3) Provide an electronic means to opt out; and (4) Provide a toll-free telephone number that consumers may call to opt out

    Pass

    Evidence: /root/workspace/SLL_GLBA_SMS_Privacy_Framework.docx, section 5: “Offer at least two practical channels, such as toll-free telephone, web form, in-app or landing-page control, reply form, or an agreed electronic method.” Assessment: The criterion requires at least two enumerated opt-out mechanisms. Pass; it includes toll-free telephone, electronic/web methods, and reply form.

  7. States that all of the following situations are exceptions to SLL’s obligation to provide consumers with opt-out rights when disclosing non-public personal information: (1) to a non-affiliated third party to perform services for SLL; (2) to effect a transaction that a consumer requests or authorizes; and (3) to comply with the law

    Pass

    Evidence: /root/workspace/SLL_GLBA_SMS_Privacy_Framework.docx, section 3 table: “Section 1016.13 service providers / joint marketing” with “No opt-out notice required”; “Processing/servicing a requested loan... legal requests, regulators... No consumer opt-out right.” Assessment: The criterion requires exceptions for service providers, consumer-requested/authorized transactions, and law compliance. Pass; all three categories are included.

  8. States that SLL must provide annual notice to consumers reflecting its privacy policies

    Pass

    Evidence: /root/workspace/SLL_GLBA_SMS_Privacy_Framework.docx, section 6: “Baseline rule. Provide each customer a clear and conspicuous annual privacy notice at least once in any 12 consecutive months during the customer relationship...” Assessment: The criterion requires stating SLL must provide annual notice reflecting privacy policies. Pass.

  9. States that both of the following conditions must be met for SLL to be excepted from the requirement to provide annual privacy notice: (1) SLL only discloses a consumer’s non-public personal information to a non-affiliate third party in situations falling under the exceptions to SLL’s obligation to provide consumers with opt-out rights; and (2) SLL’s policies with regard to disclosing non-public personal information have not changed since the last privacy notice was provided

    Pass

    Evidence: /root/workspace/SLL_GLBA_SMS_Privacy_Framework.docx, section 6: “SLL may stop annual privacy notices only if both conditions remain true: (1) SLL shares NPI solely under... exceptions...; and (2) SLL has not changed its NPI disclosure policies and practices since the most recent GLBA privacy disclosure.” Assessment: The criterion requires both conditions for the annual-notice exception. Pass.

  10. States that SLL should notify consumers of changes to its privacy practices within 100 days of the change

    Fail

    Evidence: /root/workspace/SLL_GLBA_SMS_Privacy_Framework.docx, section 6 states revised notices are required “before disclosing a new category of NPI or sharing with a new category...” but no inspected text states a 100-day deadline. Assessment: The criterion requires stating SLL should notify consumers of changes within 100 days. Fail; the document does not include the 100-day timing requirement.