LoRAs or even full fine-tunes would be much cheaper than that, and with some investment in the right infra could be updated regularly. And at least LoRAs can be swapped in and out cheaply, making them usable in large-scale inference providers. But there seems to be limited appetite in offering this. Both Anthropic and OpenAI no longer offer fine tuning for current models
There was a recent-ish paper[0] from Sakana AI about baking facts from a document corpus into a LoRA adapter. Claimed near perfect recall on very large needle-in-haystack testing. Haven’t tried it myself though.
Is this because of catastrophic forgetting, or is that more or less solved?
Is it, alternatively, because loading your LoRA plus someone else's can't be done in the same batch, so inference is made harder or becomes impossible to do at scale?
