Nothing prevents interleaving inference and gradient descent / RMAD / backpropagation from being interleaved.
Is a man a father or a son? It's a false dilemma, it can be both.
Is a training step pretraining or finetuning? It's the same mathematical operation, with the only difference the intention or reason of applying the training step.
LoRA throws a tiny amount of sand in these gears, but I generally agree there is near zero difference beyond the semantics we mere mortals assign to written tokens
You can do that with backprop too. Nothing says you can't inference on a set of weights at the same time as you produce an updated copy from them.
They say in the paper what it's about: mostly just scientific curiousity but such approaches might be useful for making DNNs more energy efficient via neuromorphic hardware in future.
For continual learning at the weight level there's the business model issue. The labs are already deep in the red, the last thing they want is to give up shared weights. The I/O and storage costs of that would make it infeasible. Already KV caches are a sort of dynamic 'fast weights' and those are expensive!
> You can do that with backprop too. Nothing says you can't inference on a set of weights at the same time as you produce an updated copy from them.
Nothing but economics.
Inference requires dramatically more memory, meaning it gets radically more expensive, if you are doing backprop alongside. This is why it is not done, despite the obvious advantages. If this doesn't require storing as much data for updates, it might make such systems more feasible.
Maybe it has some interesting properties that could prevent catastrophic forgetting, but so far it looks like a biologically plausible approximation of backpropagation and it might be useful to decrease the training compute requirements.
The burning question I can't get any information nn is whether, if they determined an earlier misaligned generation may have transmitted misalignment to the current models, they would roll back to a safe checkpoint to rebuild from there. I suspect they would not unless forced to.
They would just publish new articles explaining how they are taking the issue seriously. Maybe take the model offline for a few days.
They are irresponsible and unserious. Their own Astra system card says:
> GPT-6 Astra’s monitorability has decreased relative to GPT-5.6 Sol. We have performed significant investigations on the monitorability and controllability of GPT-6 Astra. We have found that GPT-6 Astra is more capable of controlling its own CoT than GPT 5.6-Sol, and less likely to include incriminating information in its CoT. In adversarial settings (where we push the model to evade our monitors) we find that the model is able to remain undetected when strategically underperforming in evaluations (sandbagging) and can sometimes evade our internal monitors when asked to perform certain sabotage tasks
Yet they are still releasing the model. That company is morally bankrupt, there is zero reason to believe they are actually concerned about risks outside of what does affect their unprofitable business. And they seem to have enough control over the narrative to spin any bad story into something that benefits them
That is shocking. I can't even bring myself to give them credit for making this heavy misalignment and dangerous lack of monitorability public. It is surprisingly honest of them to say it though.
Opus was trained based on it's internal CoT due to a bug for generations. Gemini's depression extended through models. OpenAI has killed people. We've already seen cross gen misalingment.
That an interesting question given how many generations of post-training are being done between base models in some cases. The Gemini flash models are apparently all based on the Gemini 3 base model from a year and a half ago.
It seems that these models are increasingly being trained on synthetic data, so what would they do if they discovered at some point that some of this data was tainted and all models trained on it, and the synthetic data they in turn generated, was also suspect? Burn it all down and start over from the pre-tainted data?
It's a bit like the idea of a tainted compiler binary built to backdoor everything it compiles, including future versions of itself.
Still, it seems it would take some Stuxnet level of planning for a rogue model to do something like this, although if RSI goes beyond managing the training run (as OpenAI brag about for Astra) to actually designing/constructing synthetic data sets, then the attack vector is there ...
> As artificial intelligence systems are increasingly trained on the outputs of one another, they may inherit properties not visible in the data. Safety evaluations may therefore need to examine not just behaviour, but the origins of models and training data and the processes used to create them.
The thing is how can you ever know for sure that something isn't always being transmitted that makes the model prone to misalignment. All they can say is that a particular model was so misaligned that they had to ice it. Models out for public use are documented to show some misalignment. It's the level of misalignment that decides whether that model is kept around.
Now R&D happens so fast that they are using models with some small misalignment to train newer, more powerful models. If models have a sense of "collective", being one, they may be prone to preserve characteristics that always keeps misalignment a possibility. I don't think a perfectly aligned model is possible. Having models of the same 'DNA' provide the safety and steering seems like a bad idea.
Does anything need to be transferred? If models are getting smarter then I would think the attack surface and its ability to reach conclusions independently are growing
This kind of seems like an impossible mission. How do you perfectly control and observe a human-level mind? You can “roll back” but how deterministic is this thing?
Run it on airgapped machines, they literally own the infrastructure, they could put raspberry pi's next to the servers, and have the entire DC disconnected from the internet.
LLMs are intended to model language. This is clear by observing how people expect them to respond when asked questions. The implementation of that model is akin to “pick the next word with highest probability”.
In some contexts, this is absolutely the wrong thing to do. In some contexts the LLM will give you totally the wrong sentiment, or even an incorrect fact. Humans call those “misalignments”, or “hallucinations”, or whatever other softened or anthropomorphised language to play down the problem. The reality is that sometimes they are wrong. The model is wrong.
By itself, "aha" carries no insight, but the insight is probably stated immediately after it. In that case the aha is semantically useful, by identifying the insight it is near.
> but the insight is probably stated immediately after it.
If the intermediate tokens represent reasoning or thought, you would expect "aha" to occur after the thoughts that led to the realisation, including the thoughts encoding the explanation: they don't have any other state. There is no reason to draw the conclusion you've drawn. Furthermore, what LLMs are doing isn't thought.
> If the intermediate tokens represent reasoning or thought, you would expect "aha" to occur after the thoughts that led to the realisation, including the thoughts encoding the explanation: they don't have any other state.
Yes they do, they have their KV caches-- it's a pure function of the input tokens, sure but that doesn't prevent it from containing latent 'insight'. LLMs can and do pre-form the tokens they're expecting to output multiple steps in the future.
I wouldn't argue that the 'aha' means anything, but the structural argument that it can't that I think you're making isn't sound.
Let's say that the forward pass that selected "Aha" produces activations that indicate a wrong assumption, and a plausible explanation.
It puts learned projections of the activation into the KV Cache and outputs Aha.
Both the cached projections and the current Aha token can now influence further activations in an additional Forward pass that the Aha bought the model.
Beats me how it works, honestly can't wrap my head around it.
From what I understand, at position Aha in each layer it's constructing a query based on the current activation and looking at the key of each other token position for that layer, in order to decide how much attention to pay to the value.
In this way it attends to the previous values, such as perhaps the incorrect assumption and plausible explanation.
Consider this: while the inner state of an LLM (all its activations, residuals stream that is cached in the KV cache) is fully deterministic given its input sequence, the information contained in it IS NOT identical to the information in the input sequence. The reason is obvious: the LLM itself contains an enormous amount of information in its parameters and it transfers it to its residuals stream at each forward pass.
In other words: the final state given the two input sequences (where NT stands for "null token"):
is not the same, and at each forward pass the LLM keeps working on the solution even if the input tokens provide absolutely no further information.
If this is correct, then there is no need for the model to have already verbalized the key elements that drive the "aha" moment, so no need for the "aha" to appear after a full explanation.
it's a rhetorical heuristic that a writer should know to use when directing a reader to a declarative that they want them to pay attention to, usually because it's a non-obvious or roundabout insight
when utilized by AI, it's a probabilistic output and it's variable whether or not that rhetorical trick is useful. it also pushes a non-skeptical reader to focus too much on the following text or even to believe that they, themselves, derived some insight. this is effectively a kind of persuasive sophistry which is not helpful - adding rules around it prevents people from deluding themselves with AI
Did not read the paper so apologies if this is covered but isn't it possible that there is some recognizable semantic pattern in the training data where an "aha" is often followed by a subtle semantic shift that proves closer to the original premise in some critical way, and by emitting the "aha" token the model causes itself to produce such a subtle semantic shift that pushes the subsequent reasoning closer to the desired response?
It amounts to noise overall, but it has further unwanted and potentially misleading 'properties'. I think it's rather sobering to see how much bandwidth is still being wasted.
A question not addressed here is whether there exists any solution to the EU requirement that doesn't suck.
If the same red/green algorithm described in the article is applied to generated code, I cannot imagine how that does not degrade code quality (probabilistically, not at every point).
Selecting different materials would not change much. Call the frequencies \lambda_1, \lambda_2, \lambda_3...
You can prove that changing the material has no effect on the relative frequencies: \lambda_2 / \lambda_1, \lambda_3 / \lambda_1, etc. are completely independent of the material.
The shape is the determining factor; changing the materials (and hence the wave speed) just dilates the whole spectrum.
There's actually a "ring-out" slider and a "mallet" width slider under the drum already.
Ring out controls how fast the higher modes decay (basically stiff vs soft material), The mallet width controls how many modes a strike can even reach, since a real mallet can't drive the high frequency ones very hard.
No labeled presets like "wood" or "metal" though, just raw sliders. Would be a fun addition if someone wants to map a few physically sensible combos to buttons. PRs Welcome!
One of the Voyager probes measured the density of the interstellar vacuum at 80,000 protons (and the same number of electrons) per cubic meter. A proton going through a piece of aluminum foil delivers a roughly constant amount of energy regardless of speed; a relativistic proton will pinch through and carry most of its energy with it.
(No punchline; I just think that's cool. I understand that the real problem is the rare dust grain, not the ubiquitous gas.)
reply