adaptive-classifier: Cut your LLM costs with smart query routing (cost savings demonstrated)

Exciting news! A new open-source library, adaptive-classifier, is here to revolutionize your LLM deployment cost optimization. This clever library dynamically routes queries between your models based on their complexity, continuously learning and refining its routing strategy through real-world usage.

Our tests on the arena-hard-auto dataset (using a high-cost and low-cost model with a 2x cost difference) yielded remarkable results:

• Achieved a significant 32.4% reduction in costs with adaptation enabled.
• Maintained the same overall success rate (22%) as the baseline.
• Demonstrated impressive learning capabilities, adapting successfully to 110 new examples during evaluation.
• Successfully directed 80.4% of queries to the more economical model.

This is ideal for environments with multiple Llama models (e.g., Llama-3.1-70B and Llama-3.1-8B) where cost optimization is crucial without compromising performance. The library seamlessly integrates with transformer-based models and features built-in state persistence for enhanced efficiency.

Explore the repository for implementation details and benchmark data. We eagerly await your feedback after trying it out!

Repository - https://www.php.cn/link/bbe2977a4c5b136df752894d93b44c72

The above is the detailed content of adaptive-classifier: Cut your LLM costs with smart query routing (cost savings demonstrated). For more information, please follow other related articles on the PHP Chinese website!