What static benchmarks miss
Predefined coding puzzles rarely capture the friction of multi-file refactoring or debugging messy environments. To find high-leverage improvements, I anchored our research on three questions: what users actually try to accomplish (task distribution), how models stack up head-to-head (relative preference), and the exact step where execution breaks down (failure attribution).
Capturing comparable, traceable feedback
Participants used a unified agent framework to run their own real tasks across three or four anonymized models simultaneously, submitting rankings, task categories, and failure notes. To make this seamless and auditable, I built a custom proxy service that standardized model configurations, managed anonymous routing, and logged full request/response payloads alongside execution traces across 6+ testing rounds.
From raw traces to training decisions
Cross-referencing human rankings with execution traces exposed specific scenario gaps and failure patterns, directly informing post-training data synthesis.
Scope & caveats：Task distributions reflect the specific cohort of each round (e.g., ~70% of tasks in one round involved information aggregation, which cannot be generalized blindly). Proprietary win-rate deltas are omitted here. Additionally, the custom proxy was purpose-built for evaluation telemetry rather than a commercial gateway with billing or strict SLA guarantees.