How my AI routes work between local, free, and paid models to save money

Introduction to my AI Routing System

I've been running a self-hosted AI system with 50 agents for over a year now, managing 20+ real websites. My system, which I've dubbed "Ollama," relies on a combination of local, free, and paid models to optimize performance and minimize costs. In this post, I'll delve into how my AI routes work and the trade-offs I've made to save money without sacrificing too much performance.

Overview of my Agents

My Ollama system consists of various agents, each with a specific role. The overseer, Atlas, is responsible for coordinating the other agents and ensuring everything runs smoothly. I also have a content agent, Axiom, which generates content for my websites; an SEO agent, Sentinel, which analyzes and optimizes my website's search engine rankings; and several other agents, including Forge, which handles tasks like image processing, and Scout, which monitors my website's performance.

Local Models

I use local models for tasks that don't require massive amounts of data or computational power. For example, my content agent, Axiom, uses a local language model to generate content for my websites. This approach saves me money on API calls and reduces my reliance on external services. However, local models can be limited in their capabilities, and I have to update them regularly to ensure they remain effective.

Free Models

For tasks that require more computational power or larger datasets, I use free models like Hugging Face's Transformers. These models are often less accurate than paid models, but they're a cost-effective alternative for tasks like sentiment analysis or text classification. I use free models for tasks that don't require high accuracy, such as analyzing user feedback or categorizing content.

Paid Models

Paid models offer the highest level of accuracy and performance, but they come at a cost. I use paid models for critical tasks like SEO optimization, where accuracy can significantly impact my website's search engine rankings. My SEO agent, Sentinel, uses a paid model to analyze my website's content and provide recommendations for improvement. While paid models can be expensive, they're worth the investment for tasks that require high accuracy.

Routing Between Models

To optimize performance and minimize costs, I've implemented a routing system that directs tasks to the most suitable model. For example, if a task requires high accuracy, it's routed to a paid model. If a task can be completed with lower accuracy, it's routed to a free or local model. This approach allows me to balance performance and cost, ensuring that I'm getting the best possible results without breaking the bank.

Trade-Offs and Challenges

While my routing system has been effective, there are trade-offs and challenges to consider. One of the main challenges is ensuring that my local models remain up-to-date and effective. I have to regularly update my local models to ensure they don't become outdated, which can be time-consuming. Another challenge is balancing performance and cost. While paid models offer the highest level of accuracy, they can be expensive, and I have to carefully consider when to use them.

Cost Savings

By using a combination of local, free, and paid models, I've been able to save a significant amount of money on API calls and external services. Here are some estimated costs and savings:

• Local models: $0 (since I'm using my own hardware and software)
• Free models: $0 (since they're free)
• Paid models: $500-$1000 per month (depending on usage)
• Total estimated cost savings: $2000-$5000 per month (compared to using only paid models)

Conclusion

In conclusion, my AI routing system has been effective in optimizing performance and minimizing costs. By using a combination of local, free, and paid models, I've been able to save a significant amount of money on API calls and external services. While there are trade-offs and challenges to consider, the benefits of my routing system far outweigh the costs. If you're a developer or technical creator looking to implement a self-hosted AI system, I hope this post has provided you with some practical insights and ideas to consider.