Redefining "resolution"
Developers asking about model selection, billing rules, or API errors need accurate answers they can immediately act on. Optimizing purely for fewer human handoffs risks frustrating users with vague deflections. We paired automated resolution/deflection with user satisfaction, analyzing over 5,000 real support tickets to build an evaluation baseline that mirrored actual inquiry distributions.
Navigating structure instead of matching fragments
Standard chunking strips away prerequisites and structural context. Instead, I designed the retrieval flow to traverse the documentation hierarchy level by level, retaining source paths before reading relevant sections in full. When queries about platform promotions repeatedly failed, trace analysis revealed that the underlying documentation tree was poorly labeled. Rather than patching the prompt, we restructured the upstream information architecture by adding a dedicated promotions entry.
Outcomes and boundaries
Automated resolution rose from a 78% baseline to 88% on launch day, stabilizing at 95% after six months of team iteration with satisfaction above 4.0/5.0.
Method limits：Navigation-based retrieval thrives on well-structured, authoritative documentation. For private account states, real-time telemetry, or unstructured logs, hybrid search and dedicated API tools remain essential.