If you're juggling multiple model providers, the practical pitch here is consolidation without a tax. This open-source gateway puts your self-hosted, frontier, and open-source models behind a single interface, and it normalizes the annoying differences between providers—streaming formats, tool-call syntax, parameter names, rate limits, and inconsistent error handling. That's the plumbing most teams end up rebuilding in-house.
It's written in Rust and built for concurrency, which shows up in the numbers: the team claims under 1 ms of added latency for bring-your-own-key requests and under 2 ms when the service supplies the provider key. The catalog covers every major inference provider and more than 1,000 models, refreshed daily by an agent that opens a pull request when new models appear. Practically, that means less manual maintenance to keep your model list current.
The differentiator versus similar tools is the business model and the routing layer. It's fully open source, charges no markup on tokens, and lets you blend local models with a marketplace. Their argument is straightforward—basic routing doesn't justify a 10% token surcharge—and it's a fair point if you're running high volume.
The smarter-routing claim is where it gets interesting. Using standardized OpenTelemetry traces, the system mines representative real tasks, simulates how different models would handle them, scores results with an LLM judge, and fits a nearest-neighbor classifier over prompt embeddings to pick the best model per request. The goal is a better cost/quality Pareto curve than pinning everything to one model. It won't be perfect, but as a starting heuristic it can catch cases where a cheaper model is good enough.
The same simulation data feeds other outputs: cache-hit optimization suggestions, recommendations for new models to try, and opt-in training of a model tuned to your traffic. Note the opt-in framing—your requests are only used for this if you choose it, which is worth confirming before you route production traffic.
What you can do with it: self-host on your own infrastructure for full control, or use the hosted version with zero markup if you'd rather skip the ops. Either way, start by pointing non-critical traffic through it to benchmark the latency claims and see whether the routing genuinely improves your cost-per-quality before you commit real workloads.