Everyone has a different idea of what is permissable. We can't even solve alignment amongst humans, what makes us think it is possible to solve alignment with AIs? It's irreducible complexity.
I have to wonder if some us are just much more inured to salespeople and thus also to "AI Safety" propaganda. I've yet to have a logical discussion with anyone who thinks the "AI Safety" people should be in charge and I think they just truly don't that what most them actually want is to be the one holding the keys to power.
The problem is the "AI Safety" people seem entirely focused on a sci-fi "the computer is a vengeful god" plot and not at all on the AI talking people into suicide or ruining children's educations. This makes them seem unserious and out of touch.
I don't disagree with your point, but I did read a post by Sean Goedecke[1] (whose opinions on the world of LLM-stuff I've generally come to respect) that I found relevant. It's not that the second-order effects don't matter to them, it's more that both the cat's probably out of the bag on those negative externalities regardless of the progress of frontier models, and that they are truly, sincerely, in-their-bones worried about the first thing and therefore focused on it since that's something that they might still have some agency over.
OK understood. But you understand this makes them sound like the type of person who doesn't believe any of the "worldly concerns" are worth addressing because "the end is near" right?
Climate change is inescapable so a vengeful AI god would pull that lever first since it has both a perverse incentive to generate more power and make the planet uninhabitable for humans in a way that cannot be overcome even if humanity fought back against the AI and regained its freedom.
Maybe compare the argument that AI has large detrimental environmental impacts to the argument that it has economic impacts. Why would the environmental impacts even be a major argument vs. the worldly economic concerns? Because there is climate science predicting extremely negative effects on humans from warming, e.g. "the end is near" on limiting climate damage. The environmental argument wouldn't have been reasonable to bring up in the 1950s if AI had gone according to the earliest optimistic plans and not required giant data centers.
There is quite a lot of mathematical research into agentic behavior that suggests a combination of instrumental convergence and the orthogonality thesis make it very likely a superintelligent agent will have arbitrary goals that lead it to attempting a takeover of Earth's resources to achieve them.
There can't be a science of superintelligence because it doesn't exist yet, but the best theories I have read seem sound, similar to how 19th century theories of anthropogenic climate change turned out to be sound.
And the real threats are mostly economic and environmental - the concentration of the means of production in the hands of few who use it to exploit us all, and a surge in energy usage accelerating climate change.
Even the sci-fi scenario assumes there is a discrepancy of capability between attacker or defender. If the 'attacking' system is (by some reasonable measure), 1000% as capable as a human, and the 'defending' systems are 60%, then it is a problem. If the 'attacking' system is 1000% as capable as a human, but there are hundreds of thousands of systems that are 900% as capable as a human, it's probably not going to take over everything successfully.
So unequal distribution of AI technology, and lax regulation and opacity of the biggest companies which actually make the risks the worst.
I don't think the "AI Safety" people are "unserious and out of touch" - I think they are actively making AI Safety problems worse by being advocates for consolidation of AI development and lack of transparency.
With some technologies, if you match attacker and defender effort you're safe, but not all of them. Nukes are the traditional example: if every person was a "nuclear power" we'd be dead within the day. But luckily there are bottlenecks to nuke creation that we can track.
What worries me the most (enough that I left tech to work on this full time) is bio. The amount of effort needed to defend against a pathogen can be many orders of magnitude higher than the effort needed to create it, and the upstream bottlenecks are both less limiting and mostly undefended. We've been safe so far because bio is very hard, but "rare expertise" as a gatekeeper is on its way out.
It's the problem of the banality of evil. Stopping someone from dramatically pressing a big red button doesn't solve humanity's largest problems because that's not what caused them. We need to stop millions "boring" actions done by systems blindly following instructions without regard for the consequences.
But those things fall in a category of "things that are awful and I'd like to see solved", which is different than "existential risks which could see my kids dead, and there's nothing I can personally do to shield them from it".
Doesn't it feel like the exact opposite? If you care a lot about the AI being a vengeful god, you do not do what OpenAI and Anthropic are doing. These companies only pay lip service to the idea of that aspect of AI safety in the hope that they can use it to regulate open source AI out of existence.
Instead, they are focused on stuff like "can I ask the AI to help me build a nuclear bomb" or "is the AI willing to generate pornographic stories", which is neither trying to protect us from unleashing a vengeful god NOR preventing (in any honest way) the today-level problems you (very correctly) bring up.
There’s also an element of it which is total misdirection.
We should be paying at least as much attention to the people who want to use AI to consolidate their wealth and power, and how they’re trying to do that. They’re a clear and present immediate danger to our societies, not something we can only speculate about. And if we deal with them, better control of AI will be a side effect.
Yeah, there are very real effects happening now regarding labor as well but to dismiss it all and worry about science fiction that is on par with evangelical beliefs is just extremely weird.
I'm yet to have a productive conversation with anyone who whinges about not being able to have a logical discussion about things they feel strongly about, but given that safety and control are interchangeable when the intentions are removed, this comes across as a particularly demagogue depiction of the subject matters involved.
A lack of control is not equivalent to freedom, the same way the totality of it is not equivalent to tyranny. There's a reason we have separate words for these things. This constant motivated conflation of the two is beyond grating. You're crying wolf until nobody believes you when they should. Don't go acting all surprised when that happens.
The issue is with the ownership of control, not necessarily with control. Attacking the latter sidesteps this rather than address it.
I trust the AI 2040 people, because they’ll make an implementation that at least convincingly doesn’t actually put them (or anyone) in charge.
Maybe we shouldn’t have AI safety, but if we’re going to trust anyone (besides yourself), who’s more qualified?
And it makes me suspicious when anyone brings up safety and doesn’t address the elephant in the room, that they’re at least unaware of the sprawling edge-cases.
> I've yet to have a logical discussion with anyone who thinks the "AI Safety" people should be in charge and I think they just truly don't [know] that what most them actually want is to be the one holding the keys to power.
I am an AI Safety Person and I want the government to nationalize or have a significant stake in the frontier labs and to have democratic control of the development of the technology. The AI Safety movement is not a monolith. I do not think Eliezer Yudkowsky nor his acolytes should hold the reins, but a lot of folks sure like to create a strawman that anyone who wants to regulate OpenAI is somehow an EA/MIRI weirdo
In my mind, democratic control of the technology means that we (the govt, or other empowered agency) take ownership of their assets and IP, solve or find a level of alignment or guardrails that society is comfortable with. Then we distribute the technology, or access to it, to avoid power concentration. This would also certainly require international coordination with China on a slowdown or pause, which I think is possible.
> This would also certainly require international coordination with China on a slowdown or pause
Personally I think the country with a strictly meritocratic elite selection system that also just outright kills you if you sell weed will have a hard time sympathizing with Bay Area thinkers who talk about AI killing us all during their ayahuasca breakfast before returning to their meth fueled crunch towards releasing the next version of the AI that will kill us all.
>I think the country with a strictly meritocratic elite selection system
Not really relevant to the broader discussion, but this simply isn’t an accurate description of China. Starting with the gaokao, admission quotas are set by province and admits to Peking university and Tsinghua are disproportionately from the urban professional class. Candidate party members must be politically vetted, which means that people whose families have expressed anti-communist views, are members of banned organizations (e.g. falun gong), or have substantial criminal records will not be permitted to advance. And once you make it into the party and enter political service, your advancement relies upon opaque patronage networks that someone without connections is unlikely to be able to navigate, even if they successfully satisfy the economic metrics the state assigns them.
I don’t want to overstate this, the Chinese system does filter out a lot of chaff and the current Chinese leadership has a lot of very capable people in positions of power. But I do not think it is substantially more meritocratic than Western political institutions
It is substantially more meritocratic on domains that matter for governance, your analysis of the incentive structure between systems is off.
1) this 2026, old school CCP patronage networks are broadly dismantled.
2) even in the mass patronage, mass corruption days, system selects for BOTH corruption competence AND performance competence for the simple reason a CCP bureaucrat has to start from the bottom and climb up, which means they need to be good with patronage AND they need to be good with hitting development KPIs. More meritocratic they are at doing their jobs, the higher they climbed, the more they get promoted and more $$$ to graft, because ability to graft directly tied to actual job competence. Hence even cliques/patronage network has to select for actual competence. This works in PRC because there are many people, and hence pool of competence is high, they can have BOTH corruption and competence, i.e. whatever pool they draw from is ultimately filtered by performance meritocracy due to incentive structure. There is reason why PRC only country where positive corruption levels was correlated to positive growth.
This is not the western system where any idiot can enter politics at anytime, and they only domain they need to optimize for is popularity to get votes.
CCP cadre evaluation strictly does not evaluate on popularity domain. It focuses on administration/execution and in so much it needs to focus on patronage... which btw any political system has (factions/cliques)... the patronage system still selects for execution, not popularity. On side, functionally what west politics selects for IS mass patronage (popularity), so attention meritocracy and not performance meritocracy, aka completely stupid incentive structure for governance. West also has ample, ample corruption, "legalized" under lobbying and paper pushing industries, so I suppose west also meritocratically selects, except for lawyers etc, and KPIs is # of document generated and not # of things build. The two are not the same when it comes to nation building.
You can have an arbitrarily low opinion of Trump and still trust in democracy as a form of government over autocracy or oligarchy, which is what the AI labs have.
You can have abstract trust in democracy over autocracy, but this is not an abstract question - these are the actual choices. Does it really matter if the person who will use AI to fuck us has a "democratic mandate" to do so? We're fucked either way.
You don't speak for anybody outside of your borders, so please stop trying to.
The majority of the world will refuse to be subject to American AI laws under the control of the US gov't, whether it pretends to be democratic or not. Or should.
Exactly. No matter how bad Trump is, millions of American people at least voted / vouched for him. Nobody voted for Elon Musk, Sam Altman, or any other CEO to have as much power as they have. Maybe a handful of shareholders. Totally unelected and unapproved by the public, but run companies that affect most of the public's lives.
That's an extremely fair objection! But despite the many, many flaws of our government and the current administration, I still have (hopefully) a chance to vote them out of power. I have no such hopes with Sam Altman or Elon Musk.
>but a lot of folks sure like to create a strawman that anyone who wants to regulate OpenAI is somehow an EA/MIRI weirdo
Anytime I see a random person on X who makes these kind of safety alarmism posts >90% of the time can be directly tied back to EA / LessWrong / related offshoots. You can not deny the amount of people, employees of Antrhopic / OpenAI, CEO and employees of various AI companies, etc are related to these groups.
Fair objection, but I still think it is lower risk to diffuse power and control of a potentially dangerous and revolutionary technology than to leave it in the hands of the few elites who have not show much ethical integrity so far.
Do you think things like the Manhattan Project were a mistake? Comparing AI to nuclear weapons is perhaps a stretch, but I think most people recognize that certain technologies or artifacts are best monopolized by our governing bodies. I think if the capabilities of AI systems keep growing on trend, it is not unreasonable to think wonton usage could disrupt society or cause mass harm.
There are many technologies that have killed and hurt a lot more people than nuclear weapons that are not monopolised by governments.
Cars and guns are two examples where ‘wanton usage’ can (and has) killed millions of people. Obviously they need regulation, but total control, even if it were possible doesn’t strike me as ideal.
Trump and Hegseth cannot be compared to Roosevelt and Stimson. (Notwithstanding Hegseth's illegitimate attempt to rename the DoD to what it was in Stimson's day.)
I had an extensive argument with someone on FB yesterday who was very much on the "I trust Dario most and I want him to be in charge" side.
I find it hard to accord with this world view at all or understand how it can come about. But it's especially troubling as a non-American to see people talk about "we can't risk AI falling into the hands of hostile authoritarian states" while advocating that AI be regulated and monopolized by the current US corporate/gov't mixture.. which is now taking the form of a hostile and authoritarian state w/ Silicon Valley as its bedmate.
Public trains are covered in camera surveillance and that footage is very often used in conjunction with tracking down and prosecuting criminals. You should have no expectation of privacy on a public train.
I agree we should also add live facial recognition to train stations.
No expectation of privacy in public does not mean we should eliminate privacy from public areas or allow our government to engage in mass surveillance.
> It was Levandowski who...had persuaded Google’s leadership to spend millions of dollars inventing self-driving cars.
> Some of the biggest fights involved risks that Levandowski was taking in self-driving experiments. ... “If it is your job to advance technology, safety cannot be your No. 1 concern,” Levandowski told me. “If it is, you’ll never do anything. It’s always safer to leave the car in the driveway. You’ll never learn from a real mistake.” [0]
What do you think of the current efforts to go from quarterly to semiannual corporate reporting in the way it is playing out in the current administration?
Well, given that it was the company I founded, LTSE, that made the original petition at the SEC to try to move in this direction, you can guess. At the time, everyone said it was a fool's errand. Now that it seems likely to succeed, we've mostly been written out of the story.
However, the details matter a great deal, and I don't know that much about what the actual proposal the SEC is going to adopt is going to be. The last draft that I saw left me pretty worried.
What's interesting to me is that, in all the hubbub, neither the journalists nor the policymakers seem that interested in the actual evidence that we have amassed in academia on this question. For example, one important study suggested that moving from semi-annual to quarterly reporting costs companies something like 5% of their market cap. It's incredibly expensive, not because generating the reports is expensive, but rather the evidence seems to be that companies under quarterly reporting start to run the company for the benefit of the report rather than for customers.
That this is bad for investors, I hope you will see as self-evident.
OTOH one could previously pay an Instagram support contractor to do an account swap, so having a human in the loop allows for other avenues of exploit:
Since their private equity buyout in 2022, Zendesk will absolutely attempt to get you to sign agregious terms. They will attempt to hoodwink you. They will pretend they never received your cancellations and charge you for another year of contract. Legal counsel should be leveraged when dealing with them to ensure you don't get screwed.
I would argue that PE often makes things better for the consumer in the sense that they often buy businesses that are going out of business.
When the 65 year old sole proprietor of a local HVAC business sells it to a PE firm, the other option was likely winding down the business. If the owner had children that were interested in running the business, no doubt they would give it to them. But usually that is not the case. So the owner needs to sell. And if there were capitalized, enthusiastic local entrepreneurs that could buy it, no doubt the owner would consider them. But again, this is quite rare.
So the choices usually come down to: close up shop or sell to a PE firm. All other things being equal (which they never are), I think a market with more businesses is going to be more competitive and pro-consumer than a market with fewer businesses. Further, some economists have found that PE activity encourages business formation, perhaps partially explaining why the US has more small businesses per capita than Europe (where they have far less PE). So it's a double whammy: PE causes less businesses to close and more businesses to open.
Yeah but they're boring. Hilton, Skype, and Dell are PE success stories. PE, when they're not being egregious, makes the website go down a bit less? Support improves? Security and compliance gets a bit more spent on them and they land more enterprise deals? Hardly the stuff to counteract taking down everyone's beloved Toys'Я'us with leveraged debt.
Sure, in the case of private equity rollups in HVAC and electric and plumbing you often get higher rates but professionalized systems including real contracts, phone answering, emailed invoices, real quotes, real terms, enforceable warranties. Many who have trusted vendors may view this as degradation because of the cost increase but for others (a new homeowner with a job), the professionalization is a positive.
For software like Zendesk that was already thoroughly professionalized, I agree, it's hard to think of positive attributes to a PE buyout!
reply