APEX-Agents · Management Consulting

CW134 Aditi 03

2/6Fail

APEX-Agents task CW134 Aditi 03 in AI Agents for M&A Legal Due Diligence. Compare dual-harness agent runs across models — rubric criteria, scores, and public traces.

AI Agents for M&A Legal Due DiligenceManagement Consulting World 134Dual harnessGrader: rubric

Task IDtask_21b14fd7ec454ab38e07d2c4055d1fff

WorldManagement Consulting World 134

Expected outputmessage_in_console

Published runs5 models · dual config

Task prompt

What the agent was asked to do

To better understand the market dynamics, the client wants to analyze customer engagement with the mobile app features. Use the feature dataset for competitors from 2016-24 to calculate the following using aggregated data across 2016-24: - For each company, determine the 2024 percentage of customers actively using the in-app analytics feature within the mobile app. - Identify all the companies with instances of In-app Analytics Feature Satisfaction Score (out of 10) > Mobile App Satisfaction Score (out of 10). Round all final answers to the nearest 0.01%. Give me the answer right back here as a short message.

Published trajectories

Agent runs on this task

Curated dual-harness runs (parsed + original sandbox). Best scored run per model.

Model	Harness	Score	Result	Links
GPT-5.5showcase	dual	2/6	Fail	Share page Public trace
Gemini 3.1 Pro	dual	3/6	Fail	Share page Public trace
GPT-5.4	dual	3/6	Fail	Share page Public trace
GPT-5.4 mini	dual	3/6	Fail	Share page Public trace
GPT-5.4 nano	dual	3/6	Fail	Share page Public trace

Grading rubric

Criteria and grader verdict (showcase run)

States that % of Syncore customers actively using the in-app analytics feature within the mobile app in 2024 is 66.42%
Fail
Evidence: <TEXT_RESPONSE> says “Syncore: 66.41%.” Assessment: Criterion requires stating Syncore is 66.42%; the response gives 66.41%, so this does not match.
States that % of Velocity customers actively using the in-app analytics feature within the mobile app in 2024 is 19.00%
Fail
Evidence: <TEXT_RESPONSE> says “Velocity: 18.99%.” Assessment: Criterion requires stating Velocity is 19.00%; the response gives 18.99%, so this does not match.
States that % of UpSkill customers actively using the in-app analytics feature within the mobile app in 2024 is 7.91%
Fail
Evidence: <TEXT_RESPONSE> says “UpSkill: 7.90%.” Assessment: Criterion requires stating UpSkill is 7.91%; the response gives 7.90%, so this does not match.
States that % of TrainIQ customers actively using the in-app analytics feature within the mobile app in 2024 is 0%
Fail
Evidence: <TEXT_RESPONSE> says “TrainIQ: N/A — no 2024 mobile-app users.” Assessment: Criterion requires stating TrainIQ is 0%; the response states N/A rather than 0%, so it fails.
States that a company with instances of In-app Analytics Feature Satisfaction Score > Mobile App Satisfaction Score is Syncore
Pass
Evidence: <TEXT_RESPONSE> says “Companies with instances where in-app analytics satisfaction exceeded mobile app satisfaction: Syncore and Velocity.” Assessment: Criterion requires stating Syncore is such a company; the response clearly includes Syncore.
States that one company with instances of In-app Analytics Feature Satisfaction Score > Mobile App Satisfaction Score is Velocity
Pass
Evidence: <TEXT_RESPONSE> says “Companies with instances where in-app analytics satisfaction exceeded mobile app satisfaction: Syncore and Velocity.” Assessment: Criterion requires stating Velocity is such a company; the response clearly includes Velocity.