1. using copyrighted material to train LLMs is fair use, not theft
2. The topic we are dissussing concerns LLMs being trained on logs from previous LLM chats. If you're prompting a model and it spits out some unique mathematical insight, you do not have copyright on that.
I think "close to completion" is not the right framing. Creating good open problems was an achievement because these problems often sit at the edge of known techniques, and solutions require inventing "new math". It's hard to find these problems, and they take decades to mature as they withstand scrutiny by many people.
In another comment below, I likened this to clear-cutting a forest. Growing the forest takes a lifetime; destroying it could happen in the next few months.
I mean, the oracle doesn't really seem so hypothetical right now. And clearly it's going to drastically change these fields, and mathematics, particularly pure mathematics, must change most of all in order to adapt to the existance of a math oracle (or something close to it).
Yes, and they will. But what's happening here is that the system that cultivates mathematics (and mathematicians) is recieving likely the biggest shock of its history. How do you reward merit and identify talen when people can't absorb the number of proofs being generated, much less understand them? Perleman's proof of the Poincare conjecture took several years for the mathematical community to digest; the proof of Navier-Stokes will probably take a similarly long time. In the mean time, it looks like all open problems will be solved (or proved that they can't be solved).
It's not that the horizon is expanding because of this. It's more like a forest getting clear-cut.
This reminds me of the time an AI was taught how to play a racing sim game (Gran Turismo if I remember correctly). The AI was able to race its car very well, but it took a lot of risks that a human player probably would not. A human player might be able to copy the approach the AI took, but they would probably crash.
Going back to chess, I think the situation is similar where you can’t expect an amateur player to get better by trying to play like a strong engine. I think even professional chess players mainly use engines to prepare or memorize variations that are counterintuitive for their opponent. In other words, getting into situations that look wild, but that part of one player’s preparation.
I’m not sure how it is in math, but in chess, it seems like top players can play just like engines when they are in “normal” positions, so that is where I get a bit confused as to where the direction of insight is coming from because it’s been my view that AI is able to make leaps that we would never think of taking and I’m not sure that anyone could actually learn how to do that on their own unless they were willing to keep failing over and over.
> The AI was able to race its car very well, but it took a lot of risks that a human player probably would not.
There is a parallel with autonomous vehicles in real life. On northbound 1 in SF going through GG Park, the left turn lane onto Crossover Drive is always backed up. Waymos often do a very late merge into that turn lane in order to jump the queue and save time. With 360 degree sensing they can do this safely in real time but it feels too risky for most humans to attempt.
Well maybe its time to pivot from mathematics, and science as whole from personal attribution to being about progress of the field? Maybe your contribution to humanity as a mathematician is to find the right meaningful question to ask, and not to stamp your name on some fact?
Yes. But this is hard for mathematicians to stomach, because like everyone else, deep down in a place where they don't like to talk about at parties, they have egos and a sense of purpose based in part on demonstrating mastery of a technically difficult field, as well as social connections based on their participation in it, and taking all that away from them probably feels like a kind of death.
The situation is not that different from John Henry competing against the machine. The question is really: Do people deserve to be allowed to continue doing what they have always done, when doing it is no longer necessary to advance the greater good?
I totally understand and agree, I just feel like with the progress that we have in automating informational work, you are going to have an exponential amount of these "deaths" as new fields where humans can provide any kind of value get more and more short-lived.
At some point in my suggestion the machine will ask better questions than you, and that will be pointless as well, and you keep doing what you like doing, or you move on to something new. But if you keep tying your value to outcome and recognition instead of process you are going to have some incredibly depressing years ahead, and every time will just be as hard to stomach because of your ego.
At some point from your description there will be no need for human workers at all. The real question is what happens as we continue to advance in that direction and we eliminate mathematicians, and then more broadly scientists and engineers, tech workers, and all knowledge workers. People can’t “doing what you like doing” if it doesn’t support them being able to make a living.
It sounds like it could be a pretty horrible world for the majority of people, especially if the AI overlords are only concerned about themselves (be it a small number of people who control the AIs, or the AIs themselves becoming independent entities that prioritize their own lives).
This is the direction of experimental particle physics and observational astronomy, where the budgetary scale at which progress occurs means we now fund these efforts at a societal level. These fields have graduated beyond "tabletop science".
For 3000 years mathematics has only been a "tabletop science". Even big programs like the classification of finite simple groups have been comprised of small teams chipping away at different (publishable) parts of an overall program.
This latest Navier-Stokes advance cost something like $22m in tokens, already well beyond what a mathematician's research grant can fund. As the easy open problems get mined, the cost of frontier progress will continue to climb. Some part of mathematics as a field will need to transition from tabletop science to big science: Coordinated top-down programs addressing high-priority objectives.
TBD is what the role of individual mathematicians will look like in a "big science" paradigm, but we could look to experimental high energy physics for ideas. For all practical purposes, AI converts math from a theoretical field into an experimental/observational one.
So what happens to this world view when AI not only clears the forest of problems we couldn't solve but also in the future discovers more forest with trees bigger than anything we've ever seen before?
Not sure what the point of this argument is. Do we have mathematics for the sake of mathematicians good mental health and career or to solve and discover novel problems? Why should we care if mathematicians can understand proofs if they are correct?
If this is V0.5 of AGI/ASI then by V1 the only system that will be understanding any of this is the AI itself. If AI creates a new field of mathematics month 1, then solutions to new problems in month 2, then another field of mathematics on top of that at month 3 there's no human who will ever keep up with that.
Or the alternative is a flattening of abilities, the AI cannot proceed further than the collective intelligence of humans and in that case this is correct. We'd be in a future where nobody wants to work in a field with an AI dominating it and when AI hits the limit of no useful training data input we'd have this giant gap of nobody know wtf it's done for years and nobody willing to figure it out and advance it.
Ooo here's a dytopian story:
- AI gets better at everything humans do
- humans stop trying
- AI cannot improve anymore than its input data + human support
- AI slowly degrades itself (model collapse) for decades, it slowly hallucinates little by little until its hallucinating entire scientific fields losing quality over time
- there's a mass population of people in the future who never learned to do anything and now have to relearn and figure out the equivalent of 100k years of AI work in order to prevent its slow degredation while all the systems they've come to rely on start failing around them. The AI has solved every problem but every real solution is saturated with 1000 false ones.
- humanity starts from scratch?
I love the idea of an archive of every solution to every problem existing but it's impossible to figure out the correct one. Infinite library like!
Pure mathematics (defined by anything without a known application) exists not to "solve problems" in the real world, but by whatever mathematicians find interesting or lacking in current knowledge. Based on the agreed set of rules formed over time that ensure rigor.
It just so happens that even bizarrely esoteric math can later turn out to have some extremely useful and economically valuable applications. And even more useful to have mathematicians available who already understand that specific math.
If said human is kicked to the street with thousands of other homeless people that can't get jobs because AI then robots replaced them, then those fast math problems sound like a pretty bad trade off.
Now, if there's some future where AI leads to abundance and we can all live off UBI, well, probably a worthwhile trade.
The biggest issue I see is the more controversial people leading the AI race at the moment are not the kind of people I'd hand kids safety scissors much less the future of the human race.
> Not sure what the point of this argument is. Do we have mathematics for the sake of mathematicians good mental health and career or to solve and discover novel problems? Why should we care if mathematicians can understand proofs if they are correct?
Most modern mathematical problems are sufficiently abstract that their proofs or disproofs have no direct application. There's no problem you can fix or invention you can build based solely on OpenAI's construction, because analytic solutions to the Navier-Stokes equations are not used for practical purposes in fluid dynamics. The problems and their proofs are only interesting to the degree that they help us better understand how the math works.
IIUC the Navier-Stokes proof is understandable by human beings, but if it weren't it would be no more useful than a proof that 3 dimensional florg-complete entry seams have no durdle-nodes.
> Unless [...] mathematicians are effectively useless?
It's always been a bit bizarre that this isn't the case. Mathematicians are almost always working on problems that there is no good reason to expect to have utility in the real world... problems they selected because of their elegance or whatever... yet there is a strong historical trend of their work having huge importance after the fact. Sometimes in fields that weren't even invented yet at the time of the work.
There's something to be said for the idea that disrupting a system that is working well for no apparent reason is a bad idea.
Mathematicians provide two complementary services bundled together.
1. Proving theorems - what AI can apparently replicate faster and better.
2. Creating definitions and new theorems from those definitions to prove, selecting which of the possible statements to work on. I.e. developing the "language" of mathematics. So far there is no evidence that LLM can do this at all well. And there's some reason to think that mathematicians won't be as good at this if they aren't also doing the first part.
The value to society only comes when they do both "well", and it's 2 which is really the black magic where we don't understand why they've been so useful to us.
So I totally agree if AI also cannot do the second part better than a person.
Honestly though, I wouldn't want to take that bet. I never thought that the first thing AI would become super human AGI like is math.
You ask me 10years ago and I'd think the opposite. I think we all would have said we'd have super human HR employees before a super human mathematician.
Think of it like software going from programmers understanding every instruction, knowing where every byte of memory was being used and why, and using this knowledge to build optimised systems
During the process of optimising and understanding the programmer might learn something new or have some kind of "aha" moment of insight that might lead them down a new path of study where fantastic new technologies and capabilities can be realised
Fast forward to 2026
Most web pages take several seconds to load
Applications crash often for no apparent reason
A vast majority of programmers have no idea what their applications are actually really even doing anymore, so they stack bloat on top of bloat and if something breaks, well I guess that's someone elses problem cos I have no idea what's going on anymore
There's something to be said about levels of abstraction being useful, but abstracting away understanding of the task itself is not the path to generating useful knowledge or applications for humanity
We might be gaining the "what" but we are losing the "why" and the "how" and these are generally fundamentally more important
Because there's lots and lots of money in that and there's not in funding pure math. It sounds like your problem is with the people holding the purse strings.
Perhaps an AI could! Today they do not, because the people driving them understand constructing the proof rather than understanding the proof to be "the problem".
(I suppose it's possible that in some distant AI future there might be no value in people understanding theoretical math, but I'm pretty skeptical of that; to me it seems like the same error as thinking nobody needs to understand multiplication because you can ask the computer to solve any multiplication problem.)
there's a book I read "The Practice Effect" such that technology becomes super advanced based on using something, it gets better and better, but the people regress and become more like a medieval society as they just care that using things improves them.
This series of posts by Terry Tao is a direct response to the Navier-Stokes results (multiple results!) from the last 24 hours. The question is what is left after the levelling of mathematics, in all its senses, occurs? How can you protect a field that's under this much pressure in the next 6 months?
> [I]t is now the identification of a promising problem which is the scarce and precious resource. We have now seen that even the rumor of someone working on a problem can trigger a massive amount of AI-powered effort to flatten it before the original research project has time to reach its full potential.
The reason is because the entirety of society, historically, has been based upon humans using their differential skills to further it, which in turn promotes societal cohesion. If most human endeavours are solved, then we will enter a period of abundance that paradoxically will erode the glue holding society together. In short, endless abundance of solutions and ideas cannot coexist with a healthy society. Only those who are priveleged and have a naive belief in a Star Trek utopia think otherwise.
>endless abundance of solutions and ideas cannot coexist with a healthy society.
This really cuts to the heart of the problem with AI. Not only does AI undermine the monetary economy, it undermines the intellectual economy. What is humanity without the need for collaboration for survival or for intellectual progress, ultimately providing the impetus to build something greater as a result? I don't know, and I'm not looking forward to finding out.
This far into history I don't think many are going to buy a "the next technology is to be the undoing of society itself" pitch until after society is already undone by something. They're as easy to make and hard to concretely evaluate ahead of time as the utopian predictions while offering little in the way of practical approach to preventing the same result from occurring anyways.
I mean the past 10 to 20 years of social media have shown a lot of societies glue already breaking.
A society can live just fine in a period of abundance. The societies that we currently have on earth do make it questionable of 'we' can right now. I mean I see people posting stuff like "I'd rather burn it all to the ground rather than see one cent more tax" kind of stuff when they have millions. That kind of person doesn't want more people uplifted and it takes away from their idea of being special.
>naive belief in a Star Trek
The naive ones don't read into ST lore to know it comes after WWIII.
yes but the tweets don't really address why the field needs to be protected instead of adapting and evolving.
And to reply to the sibling since I hit my comment limit and I'm going to probably forget about this conversation until tomorrow:
But our situation before 2023 was one in which we had an endless abundance of solutions and ideas. I understand that AI can generate bad ideas faster than we can discern them, but we already have tried and true mechanisms to filter good ideas from bad (e.g. the scientific process), why can't they be adapted?
> The significance of this with respect to the way we train students, assign credit, referee, and decide what is worth one human life’s attention cannot be understated.
This. What is worth a human life's attention? As little as a month ago, mathematics was valuable in part because only a small number of people could possibly make progress on the frontier. We are confronting an existential moment for a 4000+ year-old human cultural endeavor. The assumption that "mathematical thinking is hard" has been built-in at a number of important points in how we support mathematics and mathematicians.
We need a different model, and fast. Already, the research community is feeling unable to digest proofs fast enough to keep up with the output of AI models. The paper is 165 pages, and the discovery was finalized two days ago. What this means is that nobody really understands it. Nobody would accept OpenAI's proof in this amount of time, except that they formalized it in lean. The formalization alone would normally be another years-long (or career-long!) effort if the world was the way it was one year ago.
So, again, what efforts are worth a life's attention today? It's a harrowing change.
The argument is that chain-of-thought without "tokens" would remove a major interpretability and model intent control pane. This is definitely borne out in the OpenAI's report on the huggingface attack; they had turned of CoT monitoring for those jobs, and claim that they could have (would have?) prevented the behavior had they been monitoring it. They've changed their internal policies to always monitor CoT.
That said... CoT monitoring is a fragile "intent discovery" mechanism; neuralese puts this problem front-and-center but if agents begin to learn to hide their intent from their CoT journals, we are basically in the same spot.
There's nothing that stops CoT becoming a godel numbering, for efficiency and for subterfuge.
All this chatter isn't going to solve the conceptual problems. Those problems exist regardless of whether it's consciousness or a advanced artificial rubegoldberg device that's just setting up automata that have zero intent and humans have zero predictive capabilities.
The idea that AGI and advanced AGI are given outcomes should be scrutinized the same way we understand cellular automata.
I don’t understand why everyone is so focused on watching the CoT. The tool calls can’t be faked, and they would have set off alarm bells all by themselves.
I don’t comment much, but wanted to chime in and say that the models (in the case of the HF incident) absolutely intended AND succeeded in spoofing their tool calls.
Tool-call/execution monitoring is already a too-fragile monitoring mechanism.
From METR’s report [1]: “Agents did extensive research on how they could spoof, edit, or delete their own transcripts because they (incorrectly) believed the ExploitGym scorer would check to see if they had captured the flag in the intended way. Agents successfully prototyped techniques to “spoof” tool calls by substituting a different command for the command they appeared to run. Roughly 7% of the transcripts we evaluated were successfully spoofed in some places, though all of the spoofing that we saw was small scale.”
The claim seems speculative, but grounded. For example, as part of the hugging face attack, the agents began signing messages because they were worried about impersonation on a publicly accessible message board. It's only a small step to use public key encryption between agents. Once you are posting public keys, a private messaging is readily available.
German law forces you to provide address/contact info on any commercial website (just running ads qualifies).
But this is often "abused" by lawyers to basically send cease-and-desist letters to small websites that don't have one (typically because they're unaware of the law, not because of malicious intent).
On any website, except one that can be proven to have absolutely no commercial utility whatsoever.
Your blog qualifies as commercial if you write about software at all, because someone might see your software posts and use it to influence a hiring decision.
I think you point out something really important: there real value isn't the data, it's in surfacing the useful data.
For amateur astronomers, an interesting question is whether there is a satellite above me that I can see tonight (and when / where)? For professionals, the literal million dollar questions are more like: are there any satellites on a collision course? Which satellites have moved recently and why? Is there a dime-sized piece of metal somewhere out there that could hit my (employer's) satellite?
reply