There are none, this isn't a problem specific to OR as much as it is a problem with serving LLMs in general.
If you use any other meta-provider that routes your requests to third party providers, you'll likely face the same issues. If you try using any of those providers directly, you'll likely face some of the same issues as well, except you won't have the option of quickly swapping to a different one and taking your credits with you.
Extreme variance in quality and feature support per provider is probably the biggest obstacle holding back adoption of open weights models.
I've had opencode go + opencode work reliably, though I'm skeptical of how robust their data security claims are in practice because they suddenly blocked accessing Deepseek unless you were okay with the data going to China where true data privacy for something like that is illegal, which makes me wonder where it went before, which weakens my trust. It's also a lot less useful now that Deepseek is so much more expensive.
Openrouter is useful for quickly testing various models with just one API. In development, it's useful. I would not run it in production tho' for all the caveat mentioned. Go to the first party provider directly, it's cheaper usually. And the cost to rewrite to use their API is usually noting (you can even have both and a feature flag), especially if you just vibe code it.
I've been using Merge AI Gateway and it's been useful so far. They tend to add new models quickly, and support has been responsive. https://gateway.merge.dev/
I was looking for an LLM gateway and saw that the most popular one had just had a massive supply chain attack, so I wrote my own. Took about 2 weeks and initially I wrote it as a provider for pi coding agent. I connect to moonshot, qwen, Gemini, zhipu, anthropic, deepseek and OpenAI. I use models.dev to load model and pricing info. Adding new providers is pretty easy because I have a standard internal format and each provider has an adapter that translates between my standard format and that required by the provider.
Fireworks has high variance amongst models and while some are served correctly; many of them are junk / broken and degraded and it seems like they don’t even know; because even running 1k MMLU Pro questions would flag it very quickly.
I've used Requesty (https://www.requesty.ai), let's you pin down providers and build your own routing policy so you at least somewhat know what to expect.
Just use a single vendor. Literally nothing wrong with that, and you avoid the complexity of both n-1 of the vendors (leaving you with the highest quality vendor) as well as the issues with the aggregating layer.
Not sure why people are drawn to this particular blunder. The promise of vendor neutrality maybe? I'll take working product over vendor-neutral slop anyways.
They used a closed list of 3 vendors in prioritized order, and got 429ed out of two of them, while the third one stopped serving the mode.
This is less of a problem if you're running an agent locally and routing your problem to OpenRouter - you can pin to one or two models for consistency and just switch models when something goes bad. But the article is specifically about production traffic.
Never before have I heard this sentiment, NGL. Vendor-neutrality has been an OS(/FLOSS) darling for, well, the whole time.
RE:"single vendor", if this post is to believed then you might have picked one that has 100x the tool calling errors for the next SoTA model, if your single vendor serves the next SoTA in the first place. It also completely erases the notion of competition driving down prices -- that would only hurt you in the short term, but obviously would ruin the whole ecosystem long term.
How does it erase the notion of competition driving down prices? Endpoints are largely compatible, so the code change required to switch from one to another is trivial. Don't load 6 months' worth of credit in an account, keep it tight. There's fairly little lock-in.
The most significant lock-in to me isn't even something you mentioned, but rather it's model related; I personally put a little time into trying to optimize my prompts every time I change models, as they all have their own unique... flavor.
As for tool calling errors, it seems like first party providers are among the best, I got the feeling that's what he was suggesting, though of course that's why you test. You can also go directly to Together.ai or whoever else you please.
Like others have said I think Openrouter seems neat for testing, but even just as a hobbyist I've been drawn to go direct to particular providers due to irritating little issues that I now see just weren't me.
>Endpoints are largely compatible, so the code change required to switch from one to another is trivial.
Wrapping a specific implementation in a neutral function is something you learn to do in year 1 of programming.
This specific issue and argument I see in lots of different aggregator dependencies, Terraform, LiteLLM/OpenRouter.
They promise to save some hypothetical work in the future if your boss asks to change vendors, and it turns out to be very trivial work that is just a regular part of our programming job, changing a couple of lines in order to change vendor.
It's worth noting that there exists a similar set of technologies with a reasonable tradeoff, using a framework that targets different user-platforms makes sense, write-once and deploy at iOS and Android is a reasonable tradeoff, but because you are deploying to those providers simultaneously and it's a user-choice so you don't get to pick one or the other (without losing clients), there's still arguments to chosing just one and losing market share, or doubling the workload and building native for both, but this is a true engineering choice. I feel like stuff like OpenRouter and TerraForm take elements of these frontend abstraction technologies and wastefully apply them to backend tech.
A particularly egregious case is when there's an aggregation layer for aggregation layers, say, a tool that generates TerraForm or Chef configs, or a tool that generates Docker and Podman containers, or a tool that generates LiteLLM/OpenRouter configs. Sounds dumb, but it happens when there's a market share for it. Can even get to 3 layers deep.
At the foundation might be an aversion to making an irreversible choice, which is an innate emergent psychological phenomenon, but is supported by the Bezos Amazon policy of reversible and irreversible doors. But again, even if you want to be light, using some of these aggregating tools isn't necessary, you can just build on top of a tech, and switch later. The only thing you get with an aggregating layer is that the API ends up being the common denominator so you lose out on the competitive advantages of each choice, or are forced to use even more complex API logic like LLM(commonParam1, commonParam2, vendorParams= {"vendor1"=:{"vendorParam1":"blabla"}} or worse, use hard-coded aggregator provided mappings between the aggregator API and the vendor API that may be incomplete and relies on updates from the aggregator dev.
I think OpenRouter’s value proposition is less about avoiding two hour developer tasks and more about having a single place to establish policy controls and dynamic selection based on current pricing and performance data. If you can’t actually do that - for the reasons described in the article you end up pinning - they can’t deliver. But it seems to work for some use cases.
Still the best way to ensure such policy control is to have 1 provider, tops 2 or 3.
Having a router thing that reroutes to 18 different vendors is of course no way to ensure any policy control, you can add all the internal buttons and dials on policy control and ISO and GDPR compliance, but all it will do is (incorrectly) check compliance box and increase compliance risk to the 18 different vendors.
In practice most openrouter users look for the cheapest vendor, and they tend to go for chinese vendors, who love to price dump and don't have the same views on contracts and IP as the west.
Who is this vendor that is consistently providing high quality inference for all families of open weight models at a competitive cost?
That's a serious questions that I really interested in the answer too. I have 25 providers included into my coding agent harness not because I care about vendor neutrality, but because I have to keep adding new ones as inference quality degrades at the providers I was using. Its quite tiresome.
it's also my opinion. Which makes me wonder : isn't there an opportunity to create a variant of rust that would make absolutely zero compromise on UB and safety at the detriment of user experience ( which we don't care about now, with AI generating the code) ?
Like programming in a kind of super strict IL. Or the opposite : super poweful, super abstract language, yet extremely strict.
But this is already happening with iPhones. Apple is touting on-device AI and only the latest phones offer the full capabilities. Newer phones will be able to run better models, so the incentive is there as soon as someone makes the killer app that only makes sense when the model is running locally on your phone.
something i've just realized : today long-standing maths problems are falling. It's great intellectually but won't probably have an immediate impact on our lives.
Now, what will happen once long-standing physics ( and chemistry and biology) problems will start to fall and at the same rate ?
Then we're going to enter a totally different world.
It's hard to see problems in those fields falling at anywhere the same rate as math, because they are all experimental fields.
There may be some problems of type type "why does X happen?" that appear answerable in terms of known science, but even these would need verification. If you want to make advances in fundamental physics, then a promising AI-generated theory might take a decade and billions of dollars to prove or disprove.
Math is a rather unique field in being entirely theoretical, axiomatic and self-referential. It is basically the best possible case not just for AI to advance without needing experimental verification, but also specifically for today's AI technology of auto-regressive LLMs and RL training, whereby valid reasoning steps learnt in one context will also be valid in another context (i.e. there is some generalizability of learnt reasoning) as long as you have learnt the pertinent aspects of that context that the validity depends on.
For example, protein folding has been figured out by AI. This was a very big moment for science and yielded a nobel prize. Alphafold 1 happened 4 years before the first version of ChatGPT and the LLM craze we see today.
But yeah, most problems in physics, chemistry or biology require labs on top of actual hard thinking. You need to be able to design experiments in a certain way. Once you have the funding, the right tools, the right people to use those tools, then you can use LLMs to increase the speed of the calculations and so on.
Not really. These same techniques fall flat on their face when applied to most physics and chemistry problems. All of academia has already been doing ML4Science for the last 8 years. God knows how many billions have been spent.
The only two major highlights are weather modeling and folded protein backbone prediction.
Mostly everything else, either lacks enough data, or there are contraits on the size of the foundational models that render them impractical or they just fail to generalize.
Connect the AI to a fully automated lab and it may be able to generate and harvest those necessary data.
It may also produce Supercovid in the process, so...
We don't have to worry about new math as much as we do about new biology. Although "breaking all current encryption" would have some nasty consequences as well.
It's about propping up the yen in foreign exchange terms, so the essential operation is buying yen with dollars to increase the price of the former in a standard microeconomics way.
US bonds happen to be the assumed-safe sink where central banks store their dollars for this eventuality.
Currency interventions never work. They buy a bit of time maybe, but without any fixes to the underlying causes that made the intervention necessary, its a temporary solution.
Where did you learn that? It doesn't reflect the structural volumes present. Central banks make interventions all the time in line with little stabilisation programs. Those are almost always deemed success.
Maybe you deduced it by yourself? If so, fx is weird despite traditionally being seen as the simplest area in finance. E.g. It's counter-intuitive but we tend to think trade make up most FX volume globally. It's in the area of less than 3%. The majority by far is speculative and hedging.
The other trap is fx volume, people assume the know what volume is but then they learn expressions of fx volume is almost always actually tick volume.
It works very well in rich countries, but it is expensive, especially if the volume of exchanged currency is low. This is one of the reason behind the Euro, countries realised it would be way less expensive to defend the currency if they banded together. This is also a big why ex french colonies keeps colonial money, because that currency is pegged to the Euro and that peg protect it (the only way to get out cheaply is to do a West African Union or something similar, which is why anti-colonialists are big on the subject. They are right).
This is literally what central banks do. Central banks influence the economy with monetary policy, including things like this. That's all they're allowed to do, since they're intentionally not a political institution and have no legislative or executive authority. If you want the root cause to be handled, that's up to the legislative body. That's not the central bank's job.
Never developped a driver, but doesn't that require running the OS in some special mode, or in a VM, requiring a ton of time waiting for the thing to reboot over and over again ?
never understood why people say that: the syntax for defining code seems quite different from the syntax defining data structure. There's no homoiconicity in javascript..
And that's just syntax, it doesn't give you a programming language at all.
JS is a dynamically typed language with prototypical inheritance objects that work like universal key-value maps for the most part. It is also mutable.
Clojure is a dynamically typed language with key-value maps. It is also immutable.
You can surely see where I'm going , the underlying semantic model is the meaningful part. Homoiconicity doesn't give you anything special if your language can parse itself and can eval code. It just makes these completely abstract implementations simpler.
> Homoiconicity doesn't give you anything special if your language can parse itself and can eval code. It just makes these completely abstract implementations simpler.
Well, in a way it does give you something: By making expression of things like macros simpler, it makes them sometimes worthwhile, and makes it a reasonable request to have this kind of meta programming in your language at all. Without homoiconicity such things become even more difficult endeavors and often unjustifiable for the language design and its implementation.
I mentioned in other comment that homoiconicity doesn't necessarily make writing macros simpler. It makes writing trivial toy examples simpler.
But let's compare it to a modern macro system like rust's or scala's, where you get a typed object representation of the AST, and for anything non-trivial you are better off with this latter.
Also, arguably the best is to have certain features in the language itself, that can be used to build proper abstractions - so you don't have to resolve to using macros in its place.
> Also, arguably the best is to have certain features in the language itself, that can be used to build proper abstractions - so you don't have to resolve to using macros in its place.
I don't agree, because that would mean, that the language must be huge, or grow huge over time, or alternatively be extremely abstract at its core, to be able to fit every use-case. Furthermore, so far I have not seen a language, in which the language designers managed to pull it off, so I tend to think that what already has been successfully pulled off, which is macros for language extensions, is the way to go.
Also this seems to be arguing from a limiting idea about what macros do. Macros are not always the right solution for any problem, in fact often they are not, but there are things you simply cannot do otherwise (without syntactic clutter), like for example changing the order of evaluation.
I also don't agree with your point about only making toy examples easier to write. For example I have written macros for implementing new define forms, which allow to specify contracts for function arguments and return values, or a macro for automatically defining functions that communicate to API routes, based on the function name, which I used to implement a proof of concept docker client for Scheme. Those are not toy examples, but real world applications, where a small macro can have big effect.
If you think macros need to be big and elaborate and complicated and otherwise are toys, then you don't really understand the power of macros. One of my favorite macros is the following threading macro:
Small, but a great addition to the code, that improves readability in many places of the code. It doesn't have to be big or long, in order to not be a toy example, but actually be a useful macro.
Well, my main point is not to never use macros, but that in certain cases a language-native feature that can also solve a problem you would use a macro for, it's probably better (better debugging, error messages, etc).
And my other point was macro systems in other languages, which I believe are better, in part due to not having homoiconicity, like Scala or rust.
Not a really nice post. It reads like someone refusing to accept bitter feelings (which is absolutely understandable given the circumstances), and then go on to personally attacking the person who hurt you.
He should have kept the "horrible manager" parts away, as it really doesn't have anything to do with the point of the post.
reply