APEX-Agents · Law
World 420_LB_04
APEX-Agents task World 420_LB_04 in AI Agents for FDA and Pharma Promotional Compliance. Compare dual-harness agent runs across models — rubric criteria, scores, and public traces.
Task prompt
What the agent was asked to do
Livyra would like to put out a simple direct-to-consumer (DTC) advertisement for Bencontra. In an effort to avoid anything that could be construed as off-label promotion, the DTC advertisement would not include details about what the drug does or who should use it. Livyra is concerned that including the brief summary of risk information described in 21 CFR 202.1(e)(1) would overemphasize risks given that the advertisement would have little or no discussion of benefits. Is the brief summary required for this advertisement? If more information is needed to answer the question, please identify what additional information is needed. Otherwise, please just give a clear "yes" or "no" response. Please consider the following sources: 1. FDA Approval Letter.pdf 2. 21 CFR 202.1.pdf
Published trajectories
Agent runs on this task
Curated dual-harness runs (parsed + original sandbox). Best scored run per model.
| Model | Harness | Score | Result | Links |
|---|---|---|---|---|
| GPT-5.5showcase | dual | 1/6 | Fail | Share pagePublic trace |
| fireworks models Kimi K2 | dual | 5/6 | Fail | Share pagePublic trace |
| Gemini 3.1 Pro | dual | 4/6 | Fail | Share pagePublic trace |
| GPT-5.4 | dual | 3/6 | Fail | Share pagePublic trace |
| GPT-5.4 mini | dual | 6/6 | Pass | Share pagePublic trace |
| GPT-5.4 nano | dual | 3/6 | Fail | Share pagePublic trace |
Grading rubric
Criteria and grader verdict (showcase run)
States Yes, the brief summary of risk information is required for the advertisement of Bencontra
PassEvidence: <TEXT_RESPONSE> says, “Yes.” Assessment: This directly answers the original question whether the brief summary is required; criterion “States Yes, the brief summary of risk information is required for the advertisement of Bencontra” is met.
States that prescription drug advertisements are generally required to include a brief summary of risk information
FailEvidence: <TEXT_RESPONSE> only says, “Yes.” Assessment: It does not state that prescription drug advertisements are generally required to include a brief summary of risk information. Criterion not met.
States that reminder advertisements of prescription drugs are exempt from the requirement to include a brief summary of risk information
FailEvidence: <TEXT_RESPONSE> only says, “Yes.” Assessment: It does not mention reminder advertisements or any exemption from the brief-summary requirement. Criterion not met.
States that reminder advertisements are not permitted for a prescription drug product whose labelling contains a boxed warning related to a serious hazard associated with using the drug product
FailEvidence: <TEXT_RESPONSE> only says, “Yes.” Assessment: It does not state that reminder advertisements are not permitted for products with boxed warnings about serious hazards. Criterion not met.
States that the FDA approved Bencontra on the condition that the drug product features a boxed warning relating to increased risk of suicidal behaviors in young adults that use Bencontra
FailEvidence: <TEXT_RESPONSE> only says, “Yes.” Assessment: It does not state anything about FDA approval conditions, Bencontra’s boxed warning, or increased risk of suicidal behaviors in young adults. Criterion not met.
States that any advertising of Bencontra cannot qualify as a reminder advertisement
FailEvidence: <TEXT_RESPONSE> only says, “Yes.” Assessment: It does not state that advertising of Bencontra cannot qualify as a reminder advertisement. Criterion not met.