Why full-context dumping failed
The initial baseline stuffed definitions for over 400 tools into a single prompt, exceeding 80,000 input tokens. Time-to-first-token alone often exceeded five seconds, and attention dilution across the massive prompt kept tool-calling accuracy around 50%. Because drivers cannot wait for multi-step reasoning loops, the highest-leverage intervention was shrinking the input before it ever reached the main model.
Multi-path retrieval grounded in spoken commands
Searching tool descriptions alone missed terse commands like "play music," while matching tool names failed when a phrase like "go home" actually mapped to a parameter inside a generic navigation API. I built a three-way retrieval pipeline spanning descriptions, tool names, and parameter schemas—identifying the primary functional domain first, blending in cross-domain candidates, and passing a deduplicated set of 30 tools to the main model.
Separating speed, recall, and execution
Input tokens dropped by roughly 90%, bringing end-to-end p90 latency under five seconds. Target-tool recall reached 92%, and end-to-end execution accuracy climbed from 50% to 88%.
Metric boundaries：Separating recall (92%) from final accuracy (88%) allowed us to isolate retrieval misses from model reasoning errors. Likewise, achieving p90 < 5s met our stage target but does not imply hard real-time guarantees for every tail request. Following this validation phase, the architecture was handed off to the delivery team.