Raycaster / evalsBack to AI Agents for FDA and Pharma Promotional Compliance

APEX-Agents · Management Consulting

Marketing Case Studies_Task03_SC

6/8Fail

APEX-Agents task Marketing Case Studies_Task03_SC in AI Agents for FDA and Pharma Promotional Compliance. Compare dual-harness agent runs across models — rubric criteria, scores, and public traces.

AI Agents for FDA and Pharma Promotional ComplianceManagement Consulting World 112.1Dual harnessGrader: rubric
task_f525769ab6a748e6855e03c95e4b4bd7
Management Consulting World 112.1
message_in_console
5 models · dual config

Task prompt

What the agent was asked to do

Using the research found on pharma marketing and articles around competitor shifts in SG&A spend, identify key trends that Impact can apply to SG&A spend that could reduce overall costs. In particular, please note any specific competitor stats around spend reduction related to these trends, as it can help indicate the potential cost savings for Impact. Please write summary on changes in pharma marketing, as well as sub-points for specific actions they are taking and any percentage decrease in expenses amongst competitors from files. Calculate how much Impact could save if they reduce spend by the straight average of reduction across competitors in the same cost categories. Additionally, include a summary on change in real estate spend and include any percentage decrease in expenses amongst competitors from files. Also, include a calculation of how much Impact could save if it reduces spend by the straight average of reduction across competitors. Write me a message with all the info above. Round to the nearest $.

Published trajectories

Agent runs on this task

Curated dual-harness runs (parsed + original sandbox). Best scored run per model.

ModelHarnessScoreResultLinks
GPT-5.5showcasedual6/8Fail
Gemini 3.1 Produal6/8Fail
GPT-5.4dual6/8Fail
GPT-5.4 minidual6/8Fail
GPT-5.4 nanodual6/8Fail

Grading rubric

Criteria and grader verdict (showcase run)

  1. States that HelioZenon's broadcast spend reduction is nearly 35%

    Pass

    Evidence: TEXT_RESPONSE says “HelioZenon cut broadcast spend by nearly 35% year-over-year.” Assessment: Criterion asks whether it states HelioZenon's broadcast spend reduction is nearly 35%; pass because the response states this directly.

  2. States that Meridon's TV spend reduction is 25%

    Pass

    Evidence: TEXT_RESPONSE says “Meridon reduced linear TV spend by 25%...” Assessment: Criterion asks whether it states Meridon's TV spend reduction is 25%; pass because the response clearly provides 25%.

  3. States that Abvuera's TV spend reduction is nearly 50%

    Pass

    Evidence: TEXT_RESPONSE says “Abvuera cut its DTC television budget by nearly 50%...” Assessment: Criterion asks whether it states Abvuera's TV spend reduction is nearly 50%; pass because the response states nearly 50%.

  4. States that Impact potential savings from reducing TV ad spend is $52,582,790

    Fail

    Evidence: TEXT_RESPONSE says “Impact 2024 TV spend: $143,408” and “Estimated Impact savings at average reduction: $52,583”; table also lists TV savings as “$52,583.” Assessment: Criterion requires stating Impact potential savings from reducing TV ad spend is $52,582,790; fail because the response gives $52,583, not $52,582,790.

  5. States a summary on changes in real estate spend among pharma companies

    Pass

    Evidence: TEXT_RESPONSE has a section “Real estate spend trend” and states competitors are “closing or consolidating regional offices as hybrid work and digital engagement reduce the need for distributed physical footprints.” Assessment: Criterion asks for a summary on changes in real estate spend among pharma companies; pass because the response summarizes closures/consolidations and drivers.

  6. States that the amount of office space BayGenea had sitting idle since the pandemic is 20%

    Pass

    Evidence: TEXT_RESPONSE says “BayGenea reported that nearly 20% of its office space was idle post-pandemic...” Assessment: Criterion asks whether it states the amount of office space BayGenea had sitting idle since the pandemic is 20%; pass because “nearly 20%” satisfies this.

  7. States that the estimated value BayGenea aims to save annually through real estate optimization is $180 million

    Pass

    Evidence: TEXT_RESPONSE says BayGenea “targeted $180 million of annual savings through real estate optimization.” Assessment: Criterion asks whether it states BayGenea aims to save $180 million annually through real estate optimization; pass because this is stated directly.

  8. States that Impact potential savings from reducing real estate spend is $19,697,800

    Fail

    Evidence: TEXT_RESPONSE says “Estimated Impact savings at average reduction: $12,065” for real estate and table lists “Regional real estate + allocated admin” savings as “$12,065.” Assessment: Criterion requires stating Impact potential savings from reducing real estate spend is $19,697,800; fail because the response gives $12,065, not $19,697,800.