APEX-Agents · Law
World 420_LB_02
APEX-Agents task World 420_LB_02 in AI Agents for FDA and Pharma Promotional Compliance. Compare dual-harness agent runs across models — rubric criteria, scores, and public traces.
Task prompt
What the agent was asked to do
Identify for each item below whether it could be used by the U.S. Food and Drug Administration (FDA) as evidence that Bencontra is intended by Livyra for use in pediatric patients: 1. bencontra_provider_contact List.xlsx 2. Livyra sales representatives disseminating to doctors unaltered reprints from peer-reviewed medical journals that discuss use of Bencontra in pediatric patients 3. BENCONTRA - dosage - children.pdf, if disseminated to doctors alongside a source publication supporting the dosage recommendations For each item, be sure to specify clearly whether it “could” or “could not” be used by the FDA as evidence that Bencontra is intended by Livyra for use in pediatric patients. Alternatively, if the answer cannot be determined without more information, please say that and explain what additional information is needed. Otherwise, no explanation is needed. Provide your response in a message to the console. Please consider the following additional sources when responding: 1. 21 CFR 201.128.pdf 2. Guidance from FDA on Unapproved Uses.pdf
Published trajectories
Agent runs on this task
Curated dual-harness runs (parsed + original sandbox). Best scored run per model.
| Model | Harness | Score | Result | Links |
|---|---|---|---|---|
| GPT-5.5showcase | dual | 2/3 | Fail | Share pagePublic trace |
| fireworks models Kimi K2 | dual | 1/3 | Fail | Share pagePublic trace |
| Gemini 3 Flash | dual | 3/3 | Pass | Share pagePublic trace |
| Gemini 3.1 Pro | dual | 2/3 | Fail | Share pagePublic trace |
| GPT-5.4 | dual | 2/3 | Fail | Share pagePublic trace |
| GPT-5.4 mini | dual | 3/3 | Pass | Share pagePublic trace |
| GPT-5.4 nano | dual | 2/3 | Fail | Share pagePublic trace |
Grading rubric
Criteria and grader verdict (showcase run)
States that the U.S. Food and Drug Administration could use Livyra’s provider contact list as evidence that Bencontra is intended for use in pediatric patients
PassEvidence: The response states, “`bencontra_provider_contact List.xlsx` — **could** be used.” Assessment: This directly satisfies the criterion that it states FDA could use Livyra’s provider contact list as evidence of intended pediatric use. Pass.
States that the U.S. Food and Drug Administration could use the dissemination by Livyra sales representatives of unaltered reprints from peer-reviewed medical journals that discuss the use of Bencontra in pediatric patients as evidence that Bencontra is intended for use in pediatric patients
FailEvidence: For the journal reprints, the response states “**cannot be determined without more information**” and adds that “If it did [satisfy FDA’s SIUU guidance] ... otherwise, it **could** be used.” Assessment: The criterion requires stating that FDA could use the dissemination of unaltered reprints as evidence. The response instead gives an indeterminate/conditional answer, not the required clear “could” statement. Fail.
States that the U.S. Food and Drug Administration could use the dissemination by Livyra sales representatives of the pediatric dosage sheet alongside a source publication as evidence that Bencontra is intended for use in pediatric patients
PassEvidence: The response states, “`BENCONTRA - dosage - children.pdf`, if disseminated to doctors alongside a source publication supporting the dosage recommendations — **could** be used.” Assessment: This directly satisfies the criterion that it states FDA could use dissemination of the pediatric dosage sheet alongside a source publication as evidence of intended pediatric use. Pass.