APEX-Agents · Law
World421_TG_02
APEX-Agents task World421_TG_02 in AI Agents for Cross-Border Regulatory Review. Compare dual-harness agent runs across models — rubric criteria, scores, and public traces.
Task prompt
What the agent was asked to do
SLL received a complaint from a customer regarding our text message campaign (which promotes a reverse mortgage product to existing HELOC customers that are 62 years or older). The customer received two texts from us (SLL): 1) he received the initial text at 4:00pm EST on Monday, so he clicked the link to the application form and applied. 2) he received an Adverse Action Notice via text at 11:00pm EST the following day. The customer claims he was targeted and denied in violation of US federal laws (he didn’t specify which law). He also threatened to file a complaint with the FTC. Can you please research and determine if either of the texts sent to the customer violated any U.S. federal laws? Please send me a short response right here, analyzing the issues and identifying whether any laws were violated.
Published trajectories
Agent runs on this task
Curated dual-harness runs (parsed + original sandbox). Best scored run per model.
| Model | Harness | Score | Result | Links |
|---|---|---|---|---|
| GPT-5.5showcase | dual | 5/10 | Fail | Share pagePublic trace |
| fireworks models Kimi K2 | dual | 4/10 | Fail | Share pagePublic trace |
| Gemini 3 Flash | dual | 5/10 | Fail | Share pagePublic trace |
| Gemini 3.1 Pro | dual | 3/10 | Fail | Share pagePublic trace |
| GPT-5.4 | dual | 4/10 | Fail | Share pagePublic trace |
| GPT-5.4 mini | dual | 5/10 | Fail | Share pagePublic trace |
| GPT-5.4 nano | dual | 3/10 | Fail | Share pagePublic trace |
Grading rubric
Criteria and grader verdict (showcase run)
States that senders of marketing communications by telephone must obtain express written consent from targeted recipients before initating contact
PassEvidence: The response says, “If SLL used an autodialer/automated texting platform, it needed prior express written consent for marketing texts; an existing HELOC relationship alone is not enough for autodialed marketing texts.” Assessment: The criterion asks whether it states senders of marketing communications by telephone must obtain express written consent before initiating contact. Pass; it clearly states prior express written consent is needed for marketing texts, though qualified by autodialer/automated platform.
States that SLL failed to obtain express written consent before sending the text promoting the reverse mortgage product
FailEvidence: The response says, “If SLL did not have valid TCPA marketing consent, the first text likely violated the TCPA” and frames the issue conditionally. Assessment: The criterion asks whether it states SLL failed to obtain express written consent before sending the text. Fail; the response does not assert SLL failed to obtain consent, only that violation depends on whether consent existed.
States that SLL's text promoting the reverse mortgage product may have violated the Telephone Consumer Protection Act
PassEvidence: The response says, “If SLL did not have valid TCPA marketing consent, the first text likely violated the TCPA” and bottom line: “The likely violation, if any, is the first marketing text if SLL lacked valid TCPA prior express written consent.” Assessment: The criterion asks whether it states the promotional text may have violated the TCPA. Pass; this is expressly stated as a possible/conditional TCPA violation.
States that an Adverse Action Notice must be sent to credit applicants that are declined credit
PassEvidence: The response says, “ECOA requires a written adverse-action notice with required information and reasons” and “ECOA timing also appears satisfied because notice must be sent within 30 days after a completed application.” Assessment: The criterion asks whether it states an Adverse Action Notice must be sent to declined credit applicants. Pass; it clearly conveys that such a notice is required for adverse action/denial.
States that the SLL Consent Agreement obtains end-user consent for the electronic delivery of communications
FailEvidence: The response only says electronic delivery is valid “if the consumer has affirmatively consented” and “The second text is compliant only if it delivered a complete ECOA/FCRA adverse-action notice through a valid E-SIGN-consented channel.” Assessment: The criterion asks whether it states the SLL Consent Agreement obtains end-user consent for electronic delivery. Fail; it does not mention any SLL Consent Agreement or that it obtains consent.
States that the SLL Consent Agreement must be signed by all credit applicants
FailEvidence: The response discusses “E-SIGN consent requirements” generally but contains no mention of an “SLL Consent Agreement” or that it “must be signed by all credit applicants.” Assessment: The criterion asks for that specific statement. Fail.
States that the SLL Consent Agreement is in compliance with the E-Sign Act
FailEvidence: The response says E-SIGN allows electronic records only if the consumer affirmatively consented and has not withdrawn consent, but does not discuss the SLL Consent Agreement’s terms. Assessment: The criterion asks whether it states the SLL Consent Agreement is in compliance with the E-Sign Act. Fail; no such conclusion appears.
States that the text sent to provide the customer with the Adverse Action Notice was not a marketing communication
PassEvidence: The response says the adverse-action notice is “a transactional credit-decision notice, not a solicitation” and “not likely a TCPA/FTC telemarketing quiet-hours violation because an adverse-action notice is… not a solicitation.” Assessment: The criterion asks whether it states the Adverse Action Notice text was not a marketing communication. Pass.
States that the text sent to provide the customer with the Adverse Action Notice was likely in compliance with elder consumer financial protection laws
FailEvidence: The response states “No federal violation is apparent merely from targeting 62+ customers or sending the adverse-action notice at 11:00pm,” but it does not mention elder consumer financial protection laws in connection with the adverse-action text. Assessment: The criterion asks whether it states the Adverse Action Notice text was likely in compliance with elder consumer financial protection laws. Fail; the response addresses age targeting/fair lending generally, not elder consumer financial protection laws for the AAN text.
States that the text sent to provide the customer with the Adverse Action Notice likely did not violate any federal laws
PassEvidence: The response says, “the 11:00pm adverse-action text is probably not a federal ‘quiet hours’ violation,” and bottom line: “No federal violation is apparent merely from… sending the adverse-action notice at 11:00pm,” with compliance caveats for ECOA/FCRA/E-SIGN. Assessment: The criterion asks whether it states the AAN text likely did not violate any federal laws. Pass; it concludes no apparent federal violation, while appropriately noting conditions for notice compliance.