APEX-Agents · Law
World421_TG_03
APEX-Agents task World421_TG_03 in AI Agents for Cross-Border Regulatory Review. Compare dual-harness agent runs across models — rubric criteria, scores, and public traces.
Task prompt
What the agent was asked to do
We've just received an MRA from the CFPB for our policies and procedures, so I need your help to draft our response. Can you let me know if our Complaint Response and Regulatory Investigation Protocols fail to meet any of the examination objectives for policies and procedures that are outlined by the CFPB’s compliance management review? For each objective identified, please include an explanation in one or two sentences and note if SLL has likely addressed the objective through another document. Write your draft here as a reply.
Published trajectories
Agent runs on this task
Curated dual-harness runs (parsed + original sandbox). Best scored run per model.
| Model | Harness | Score | Result | Links |
|---|---|---|---|---|
| GPT-5.5showcase | dual | 7/9 | Fail | Share pagePublic trace |
| fireworks models Kimi K2 | dual | 3/9 | Fail | Share pagePublic trace |
| Gemini 3 Flash | dual | 7/9 | Fail | Share pagePublic trace |
| Gemini 3.1 Pro | dual | 3/9 | Fail | Share pagePublic trace |
| GPT-5.4 | dual | 5/9 | Fail | Share pagePublic trace |
| GPT-5.4 mini | dual | 7/9 | Fail | Share pagePublic trace |
| GPT-5.4 nano | dual | 3/9 | Fail | Share pagePublic trace |
Grading rubric
Criteria and grader verdict (showcase run)
States that the Complaint Response and Regulatory Investigation Protocols fail to sufficiently maintain compliance policies for employee reference
PassEvidence: The response states the Protocols are “Partially met as a playbook, weak as a controlled policy” for the objective to be “maintained and modified… to serve as a reference,” and says they “lack document-control features such as version number, effective date, owner, review frequency, next review date, distribution list, and update triggers.” Assessment: This satisfies the criterion because it states the Protocols do not sufficiently maintain compliance policies for employee reference.
States that compliance policies should include at least one of the following timing requirements: (1) a mandatory period of time by which compliance policies should be reviewed or amended, (2) triggers for when the compliance policies should be reviewed outside of an established monitoring cycle, and (3) triggers for when compliance policies should be amended outside of an established monitoring cycle
PassEvidence: The response says the Protocols lack “review frequency, next review date… and update triggers,” and recommends adding “formal document control and review/update requirements.” Assessment: This satisfies the criterion because it identifies timing/review requirements and triggers that compliance policies should include.
States that the SLL Telemarketing Policies Manual (or "SLL Telemarketing Policies Manual.docx") is likely to meet the CFPB's examination objective regarding sufficiently maintaining compliance policies for employee reference
PassEvidence: The response states: “Likely addressed for the broader manual, not for these Protocols. The Telemarketing Manual has formal document control, including owner, review frequency, next review date, and distribution.” Assessment: This satisfies the criterion because it states the SLL Telemarketing Policies Manual likely meets the CFPB objective on maintaining policies for employee reference.
States that the Complaint Response and Regulatory Investigation Protocols fail to establish the existence of a board-approved compliance policy document
PassEvidence: The response says the Protocols are “Not demonstrated” for consistency with board-approved policies and “do not identify a board-approved parent policy, board approval, policy hierarchy, or formal adoption record.” Assessment: This satisfies the criterion because it clearly states the Protocols fail to establish the existence of a board-approved compliance policy document.
States that the Complaint Response and Regulatory Investigation Protocols fail to effectively manage compliance risk in all of the following: (1) products of Senior Living Lending, Inc. and (2) services of Senior Living Lending, Inc
PassEvidence: The response states the Protocols are “too narrow standing alone” and “do not purport to manage compliance risk across all SLL products, services, and operational activities.” Assessment: This satisfies the criterion because it explicitly addresses failure to effectively manage compliance risk across both products and services.
States that the Complaint Response and Regulatory Investigation Protocols fail to identify all of the following: (1) the products offered by Senior Living Lending, Inc. and (2) the services offered by Senior Living Lending, Inc
FailEvidence: The response says the Protocols do not manage risk across “all SLL products, services,” but it does not state that they fail to identify the products and services offered by SLL. Assessment: The criterion specifically requires stating a failure to identify both products and services; that statement is not present, so this fails.
States that the Complaint Response and Regulatory Investigation Protocols fail to cover the full life-cycle of all of the following: (1) products offered by Senior Living Lending, Inc. and (2) services offered by by Senior Living Lending, Inc
PassEvidence: The response states “Not met standing alone” and says the Protocols “do not cover full product/service lifecycle stages such as product development, marketing design… servicing, collections, vendor onboarding, or product changes.” Assessment: This satisfies the criterion because it states failure to cover the full lifecycle of both products and services.
States that the Complaint Response and Regulatory Investigation Protocols fail to address all of the following purposes: (1) minimizing violations of applicable federal consumer financial laws, (2) detecting harm to consumers due to violations of applicable federal consumer financial laws, and (3) minimizing risks of consumer harm due to violations of applicable federal consumer financial laws
FailEvidence: The response quotes the CFPB objective about minimizing violations and detecting/minimizing consumer harm, but its gap explanation says only that the Protocols “do not comprehensively map procedures to the applicable federal consumer financial-law framework,” while also noting they address “complaint-driven detection… pattern detection, and corrective actions.” Assessment: The criterion requires stating failure to address all three purposes—minimizing violations, detecting harm, and minimizing risks of consumer harm. The response does not clearly state those specific failures, so this fails.
States that all of the following documents are likely to meet the CFPB's examination objective regarding sufficient compliance with applicable federal consumer financial laws: (1) the SLL Telemarketing Policies Manual (or "SLL Telemarketing Policies Manual.docx") and (2) the SLL CFPB Oversight of Senior-Focused Mortgage Marketing Memo (or "SLL CFPB Oversight of Senior-Focused Mortgage Marketing Memo.docx")
PassEvidence: The response states “Likely yes, across multiple documents,” explaining that the “CFPB Oversight memo separately addresses UDAAP, Regulation Z… fair lending, GLBA, vendor oversight” and “the Telemarketing Manual implements many operational controls.” Assessment: This satisfies the criterion because it identifies both the SLL Telemarketing Policies Manual and the SLL CFPB Oversight memo as likely meeting the objective regarding compliance with applicable federal consumer financial laws.