Talking to my AI hands-free: local Whisper + a terminal voice cockpit

Background and Motivation

I've been running a self-hosted AI system, which I've dubbed "Ollama", for several months now. The system consists of 50 agents, each with a specific role, scheduled to run at various intervals using Python's cron library. My laptop, equipped with an RTX 4050 6GB GPU, handles the computations. The agents manage over 20 real websites, handling tasks such as content creation, SEO optimization, and analytics reporting. As the solo founder, I've been manually interacting with the agents through a terminal interface, which, while functional, can be cumbersome and time-consuming.

Whisper: The Local AI Assistant

To improve my interaction with the agents, I decided to incorporate Whisper, a local AI assistant, into my setup. Whisper allows me to communicate with my agents hands-free, using voice commands to initiate tasks, query status, and receive updates. I opted for a local installation to maintain control over my data and ensure seamless integration with my existing Ollama system. The performance trade-off was negligible, with Whisper's memory footprint being approximately 300MB, leaving ample resources for my agents to operate.

Terminal Voice Cockpit: Custom Implementation

To leverage Whisper's capabilities, I created a custom terminal voice cockpit, which I've integrated into my Ollama system. The cockpit allows me to issue voice commands, such as "Atlas, what's the current website traffic?" or "Axiom, generate a report on SEO performance." The commands are then transcribed and executed by the corresponding agents. I've also implemented a set of predefined voice commands for common tasks, like "Sentinel, initiate backup sequence" or "Forge, update website content." The voice cockpit has significantly streamlined my workflow, reducing the time spent on manual terminal interactions by approximately 30%.

Implementation Details and Trade-Offs

The implementation process involved several trade-offs. I had to balance the accuracy of Whisper's speech recognition with the computational resources available. I opted for a moderately complex speech recognition model, which achieves an accuracy of around 92% on my laptop's hardware. While this may not be optimal, it was a necessary compromise to maintain a responsive system. I've also had to adapt my agents to respond to voice commands, which required updates to their respective interfaces. The overseer agent, responsible for coordinating the other agents, now acts as a voice command dispatcher, routing commands to the relevant agents.

Current Capabilities and Future Plans

Currently, my voice-enabled Ollama system supports a range of tasks, including:

• Content creation: generating blog posts, social media updates, and product descriptions
• SEO optimization: analyzing website performance, suggesting improvements, and implementing changes
• Analytics reporting: providing insights into website traffic, engagement, and conversion rates
• System maintenance: performing backups, updating software, and monitoring system resources

I plan to expand the capabilities of my system, exploring the use of more advanced speech recognition models and integrating additional agents to handle tasks such as customer support and email management. With the foundation of my terminal voice cockpit in place, I'm excited to see where this technology will take my self-hosted AI system and the opportunities it will unlock for my business.