TRACER: Learn-to-Defer for LLM Classification with Formal Teacher-Agreement Guarantees

About this article

I'm releasing TRACER (Trace-Based Adaptive Cost-Efficient Routing), a library for learning cost-efficient routing policies from LLM traces. The setup: you have an LLM handling classification tasks. You want to replace a fraction of calls with a cheap local surrogate, with a formal guarantee that the surrogate agrees with the LLM at least X% of the time on handled traffic. Technical core: Three pipeline families: Global (accept-all), L2D (surrogate + conformal acceptor gate), RSB (Residual Sur...