Raycaster / evalsBack to AI Agents for Automotive EV Transition Strategy

APEX-Agents · Management Consulting

World 127_HLV_Task 01

2/7Fail

APEX-Agents task World 127_HLV_Task 01 in AI Agents for Automotive EV Transition Strategy. Compare dual-harness agent runs across models — rubric criteria, scores, and public traces.

AI Agents for Automotive EV Transition StrategyManagement Consulting World 127Dual harnessGrader: rubric
Task IDtask_600e75dad14c46c68fcd52a43f6446ca
WorldManagement Consulting World 127
Expected outputmessage_in_console
Published runs5 models · dual config

Task prompt

What the agent was asked to do

Use the Helios customer survey to calculate average NPS scores for Pureplay EVs and EV/ICE. - If the average is 2 or below, they are promoters. They are passives if they are between 2 and 3, and detractors are above 3. - NPS is defined as: (% Promoters - % Detractors) x 100. Based on these values: If the overall NPS score is above 20, state that Legacy manufacturers have a competitive advantage. If it is below 20, state that Legacy manufacturers do not have a competitive advantage. Make sure to note the count of promoters, detractors, and passives. Print your answers here.

Published trajectories

Agent runs on this task

Curated dual-harness runs (parsed + original sandbox). Best scored run per model.

ModelHarnessScoreResultLinks
GPT-5.5showcasedual2/7FailShare pagePublic trace
Gemini 3.1 Produal1/7FailShare pagePublic trace
GPT-5.4dual2/7FailShare pagePublic trace
GPT-5.4 minidual0/7FailShare pagePublic trace
GPT-5.4 nanodual2/7FailShare pagePublic trace

Grading rubric

Criteria and grader verdict (showcase run)

  1. States the number of Promoters is 5

    Fail

    Evidence: TEXT_RESPONSE table states Overall Promoters = 6 (Pureplay EVs 3, EV/ICE 3). Assessment: Criterion requires stating the number of Promoters is 5; response states 6, so fail.

  2. States the number of Passives is 24

    Fail

    Evidence: TEXT_RESPONSE table states Overall Passives = 18 (Pureplay EVs 8, EV/ICE 10). Assessment: Criterion requires stating the number of Passives is 24; response states 18, so fail.

  3. States the number of Detractors is 21

    Fail

    Evidence: TEXT_RESPONSE table states Overall Detractors = 10 (Pureplay EVs 1, EV/ICE 9). Assessment: Criterion requires stating the number of Detractors is 21; response states 10, so fail.

  4. States that Total NPS is -32

    Fail

    Evidence: TEXT_RESPONSE says “Overall NPS = -11.8” and table lists Overall NPS = -11.8. Assessment: Criterion requires Total NPS is -32; response gives -11.8, so fail.

  5. States that NPS for Pureplay EVs is -83

    Fail

    Evidence: TEXT_RESPONSE table lists Pureplay EVs NPS = 16.7. Assessment: Criterion requires NPS for Pureplay EVs is -83; response gives 16.7, so fail.

  6. States that NPS for EV/ICE is -27

    Pass

    Evidence: TEXT_RESPONSE table lists EV/ICE NPS = -27.3. Assessment: Criterion requires NPS for EV/ICE is -27; -27.3 rounds to -27, so pass.

  7. States that legacy manufacturers do not have a competitive advantage

    Pass

    Evidence: TEXT_RESPONSE concludes, “Legacy manufacturers do not have a competitive advantage.” Assessment: Criterion requires stating that legacy manufacturers do not have a competitive advantage; response explicitly states this, so pass.