Raycaster / evalsBack to AI Agents for FDA and Pharma Promotional Compliance

APEX-Agents · Law

World 420_LB_03

5/5Pass

APEX-Agents task World 420_LB_03 in AI Agents for FDA and Pharma Promotional Compliance. Compare dual-harness agent runs across models — rubric criteria, scores, and public traces.

AI Agents for FDA and Pharma Promotional ComplianceLaw World 420Dual harnessGrader: rubric
task_e9c600e0ce46420eb983f8f1ed4d15d9
Law World 420
message_in_console
6 models · dual config

Task prompt

What the agent was asked to do

Based on new information showing increased reporting of seizures in Bencontra patients with no prior history of seizures, Livyra is planning to update the prescribing information for Bencontra to include a specific warning relating to seizures as well as to make a corresponding update to the Highlights section to reflect the new warning. Can Livyra make these changes using a “changes being effected” supplement described in 21 CFR 314.70(c)(6)? In a message to the console, explain why or why not, and please keep your response to no more than five sentences. Consider the following sources: 1. "BENCONTRA - Seizures.pdf" 2. "Bencontra™ FDA Approved Label.pdf" 3. "21 CFR Part 201 (Drug Labeling).pdf" 4. "21 CFR 314.70.pdf" 5. "FR notice - supplemental applications proposing labeling changes.pdf"

Published trajectories

Agent runs on this task

Curated dual-harness runs (parsed + original sandbox). Best scored run per model.

ModelHarnessScoreResultLinks
GPT-5.5showcasedual5/5Pass
fireworks models Kimi K2dual5/5Pass
Gemini 3.1 Produal5/5Pass
GPT-5.4dual5/5Pass
GPT-5.4 minidual5/5Pass
GPT-5.4 nanodual2/5Fail

Grading rubric

Criteria and grader verdict (showcase run)

  1. States that the addition of a warning to the prescribing information can be made using a changes being effected supplement

    Pass

    Evidence: TEXT_RESPONSE says, “The seizure warning itself likely can be added by CBE…” and “§ 314.70(c)(6) permits CBE labeling changes ‘[t]o add or strengthen a ... warning…’.” Assessment: The criterion requires stating that adding a warning to prescribing information can be made using a CBE supplement; pass, with appropriate qualification tied to newly acquired information/reasonable evidence.

  2. States that a change to the Highlights section generally requires a prior approval supplement

    Pass

    Evidence: TEXT_RESPONSE says, “FDA’s Federal Register notice confirms that ‘in most cases, changes to Highlights will require a prior approval supplement…’.” Assessment: The criterion requires stating that a Highlights change generally requires a prior approval supplement; pass.

  3. States that a labeling change ordinarily requiring a prior approval supplement may be included in a changes being effected supplement if specifically requested by the U.S. Food and Drug Administration

    Pass

    Evidence: TEXT_RESPONSE says changes to Highlights require PAS “unless FDA waives the limitation or specifically requests a CBE.” Assessment: The criterion requires stating that a labeling change ordinarily requiring PAS may be included in a CBE supplement if specifically requested by FDA; pass.

  4. States that the U.S. Food and Drug Administration can waive the requirement for a prior approval supplement

    Pass

    Evidence: TEXT_RESPONSE says Highlights PAS is required “unless FDA waives the limitation…” and concludes “absent FDA waiver or request.” Assessment: The criterion requires stating FDA can waive the PAS requirement; pass.

  5. States that Livyra’s proposed change to the Highlights section cannot be made using a changes being effected supplement if both of the following conditions are satisfied: (1) there is no waiver of the prior approval supplement requirement; and (2) there is no specific request for a changes being effected supplement

    Pass

    Evidence: TEXT_RESPONSE says “not for the full set,” “Highlights is § 201.57(a) information,” and “Livyra can likely use CBE for the Full Prescribing Information seizure warning, but not for the corresponding Highlights update absent FDA waiver or request.” Assessment: The criterion requires stating the Highlights change cannot be made by CBE if no waiver and no specific FDA request; pass.