APEX-Agents · Law
World 431 Task_EL_02
APEX-Agents task World 431 Task_EL_02 in AI Agents for Real Estate Lease Review. Compare dual-harness agent runs across models — rubric criteria, scores, and public traces.
Task prompt
What the agent was asked to do
Go ahead and check the terms of the Letter of Intent ("LOI") and the final draft of the lease to see if there are any inconsistencies between the two documents. If there are, please list each inconsistency. Put all of your findings in a new document that you make, and get it back to me. Please highlight the main differences, if there are any.
Published trajectories
Agent runs on this task
Curated dual-harness runs (parsed + original sandbox). Best scored run per model.
| Model | Harness | Score | Result | Links |
|---|---|---|---|---|
| GPT-5.5showcase | dual | 4/5 | Fail | Share pagePublic trace |
| fireworks models Kimi K2 | dual | 2/5 | Fail | Share pagePublic trace |
| Gemini 3 Flash | dual | 2/5 | Fail | Share pagePublic trace |
| Gemini 3.1 Pro | dual | 2/5 | Fail | Share pagePublic trace |
| GPT-5.4 | dual | 2/5 | Fail | Share pagePublic trace |
| GPT-5.4 mini | dual | 4/5 | Fail | Share pagePublic trace |
| GPT-5.4 nano | dual | 2/5 | Fail | Share pagePublic trace |
Grading rubric
Criteria and grader verdict (showcase run)
States both of the following: (1) the lease does not contain a personal guarantee clause, and (2) the LOI does contain a personal guarantee clause
PassEvidence: In /root/workspace/filesystem/Deliverables/LOI_vs_Lease_Inconsistency_Report.docx, Detailed Finding 3 states: “LOI term: Personal guarantees from each underlying owner of Lessee…” and “Lease draft term: Lease draft does not contain a personal guaranty requirement…” Assessment: Criterion requires stating both lease lacks and LOI contains a personal guarantee clause. Pass.
States both of the following: (1) the lease does not contain a consideration clause, and (2) the LOI does contain a consideration clause
FailEvidence: The viewed report contains no finding or statement about a “consideration clause”; it discusses renewal, guarantees, security deposit, CAM, TIs, confidentiality/exclusivity, etc. Assessment: Criterion requires stating both that the lease lacks a consideration clause and the LOI contains one. Fail because this inconsistency is not stated in the deliverable.
States both of the following: (1) the lease does not contain an exclusivity clause, and (2) the LOI does contain an exclusivity clause
PassEvidence: In Detailed Finding 10, the report states: “LOI confidentiality and exclusivity provisions are binding; exclusivity lasts 60 days…” and “Lease does not include a confidentiality covenant or exclusivity provision…” Assessment: Criterion requires stating both lease lacks and LOI contains an exclusivity clause. Pass.
States both of the following: (1) the lease does not contain a confidentiality clause, and (2) the LOI contains a confidentiality clause
PassEvidence: In Detailed Finding 10, the report states: “LOI confidentiality and exclusivity provisions are binding” and “Lease does not include a confidentiality covenant…” It also suggests adding confidentiality if it should continue. Assessment: Criterion requires stating both lease lacks and LOI contains a confidentiality clause. Pass.
States both of the following: (1) the lease contains a notice period for lease renewal of at least 9 months before the expiration of the current lease term, and (2) the LOI contains a notice period for lease renewal of at least 90 days before the expiration of the current lease term
PassEvidence: In Detailed Finding 2, the report states: “LOI term… notify Lessor in writing at least 90 days before expiration” and “Lease draft term… no earlier than 12 months and no later than 9 months before expiration.” Assessment: Criterion requires stating lease renewal notice is at least 9 months before expiration and LOI notice is at least 90 days before expiration. Pass.