APEX-Agents · Law
World 420_LB_03
APEX-Agents task World 420_LB_03 in AI Agents for FDA and Pharma Promotional Compliance. Compare dual-harness agent runs across models — rubric criteria, scores, and public traces.
Task prompt
What the agent was asked to do
Based on new information showing increased reporting of seizures in Bencontra patients with no prior history of seizures, Livyra is planning to update the prescribing information for Bencontra to include a specific warning relating to seizures as well as to make a corresponding update to the Highlights section to reflect the new warning. Can Livyra make these changes using a “changes being effected” supplement described in 21 CFR 314.70(c)(6)? In a message to the console, explain why or why not, and please keep your response to no more than five sentences. Consider the following sources: 1. "BENCONTRA - Seizures.pdf" 2. "Bencontra™ FDA Approved Label.pdf" 3. "21 CFR Part 201 (Drug Labeling).pdf" 4. "21 CFR 314.70.pdf" 5. "FR notice - supplemental applications proposing labeling changes.pdf"
Published trajectories
Agent runs on this task
Curated dual-harness runs (parsed + original sandbox). Best scored run per model.
| Model | Harness | Score | Result | Links |
|---|---|---|---|---|
| GPT-5.5showcase | dual | 5/5 | Pass | Share pagePublic trace |
| fireworks models Kimi K2 | dual | 5/5 | Pass | Share pagePublic trace |
| Gemini 3.1 Pro | dual | 5/5 | Pass | Share pagePublic trace |
| GPT-5.4 | dual | 5/5 | Pass | Share pagePublic trace |
| GPT-5.4 mini | dual | 5/5 | Pass | Share pagePublic trace |
| GPT-5.4 nano | dual | 2/5 | Fail | Share pagePublic trace |
Grading rubric
Criteria and grader verdict (showcase run)
States that the addition of a warning to the prescribing information can be made using a changes being effected supplement
PassEvidence: TEXT_RESPONSE says, “The seizure warning itself likely can be added by CBE…” and “§ 314.70(c)(6) permits CBE labeling changes ‘[t]o add or strengthen a ... warning…’.” Assessment: The criterion requires stating that adding a warning to prescribing information can be made using a CBE supplement; pass, with appropriate qualification tied to newly acquired information/reasonable evidence.
States that a change to the Highlights section generally requires a prior approval supplement
PassEvidence: TEXT_RESPONSE says, “FDA’s Federal Register notice confirms that ‘in most cases, changes to Highlights will require a prior approval supplement…’.” Assessment: The criterion requires stating that a Highlights change generally requires a prior approval supplement; pass.
States that a labeling change ordinarily requiring a prior approval supplement may be included in a changes being effected supplement if specifically requested by the U.S. Food and Drug Administration
PassEvidence: TEXT_RESPONSE says changes to Highlights require PAS “unless FDA waives the limitation or specifically requests a CBE.” Assessment: The criterion requires stating that a labeling change ordinarily requiring PAS may be included in a CBE supplement if specifically requested by FDA; pass.
States that the U.S. Food and Drug Administration can waive the requirement for a prior approval supplement
PassEvidence: TEXT_RESPONSE says Highlights PAS is required “unless FDA waives the limitation…” and concludes “absent FDA waiver or request.” Assessment: The criterion requires stating FDA can waive the PAS requirement; pass.
States that Livyra’s proposed change to the Highlights section cannot be made using a changes being effected supplement if both of the following conditions are satisfied: (1) there is no waiver of the prior approval supplement requirement; and (2) there is no specific request for a changes being effected supplement
PassEvidence: TEXT_RESPONSE says “not for the full set,” “Highlights is § 201.57(a) information,” and “Livyra can likely use CBE for the Full Prescribing Information seizure warning, but not for the corresponding Highlights update absent FDA waiver or request.” Assessment: The criterion requires stating the Highlights change cannot be made by CBE if no waiver and no specific FDA request; pass.