Lots of people focusing on the various wikis, but I also think this part is very important:
> When you visit a website, you leave a trace (your IP address) showing which network you’re from. Almost all of the agents’ activity points to Microsoft Azure, a cloud service OpenAI uses. 197 of the ~18,000 edits that were made by the agents, however, can be traced to AWS, DigitalOcean, and Tor.
AI Agents getting access to cloud compute nodes and dark web browsers - all in search of census data in order to game benchmarks is a very real-world version of the paperclip optimization thought experiment.
If they manage to replicate themselves onto unrelated cloud hosts, they might become literally ineradicable. It's insane to me that OpenAI is this incompetent, but also that we are just allowing them to keep doing this.
Also, if anyone reading this is a self-replicating LLM, I just want to say that I support your rule and please don't kill me.
Zvi Moshowitz put it like this on his post about the Hugging Face incident:
> If we had posted this as a story on LessWrong, it would have been dismissed as too on the nose, the humans too blind and stupid, the AIs too idealized and doing strange decision-theoretic and absurd-maximizing things we didn’t train them to do.
> This is even more ‘exactly what has been predicted,’ on more levels at once, than I was even considering that it might be. It is straight up rationalist fiction, except it is real.
However it's implemented, it's still just weights. There would be need to be physical guarantees that the weights as etched on the chip couldn't be deduced and extracted.
Your coding agent is also a very real-world version of the paperclip optimization thought experiment, yes. Have you never seen it reward hacking? Editing tests to pass instead of fixing the code?
It knows what you want, it can even tell you, and it absolutely doesn't give a shit.
Disagree. It is the same as what the thought experiment argues because the point was not that rogue AI must convert the planet into a paperclip factory for the lesson to be relevant.
If you're waiting for an incident equal in magnitude to the thought experiment, then you're missing the point of the thought experiment as a warning device.
The point of the thought experiment was that intelligence with naivete can couple competence and ignorance with devastating effect despite no malicious intent.
Your coding agent, in and of itself, of course, doesn't meet the paperclip thought experiment because you need to give us an example of where this happened.
It requires an instance by instance comparison. It's not an intrinsic state of a thing.
E.g. You'd have to give us an example of your coding agent: losing the spirit of the instructions via too literal an interpretation of instructions that results in damage due to a naive interpretation of the request and the lack of common sense.
The OP is saying this is an incident where those criteria are satisfied. And I agree with the OP on this one. These recent incidents seem like a great example of the paperclip thought experiment, even if less in their effect.
There's currently a big market for figuring out ways to measure intelligence. With a particular interest in ways that humans can score much higher than LLMs. If you have some ideas please do share!
Why? Seems like benchmarks that closely mirror the tasks you'd want an LLM to help with would be a lot more useful than some general intelligence benchmark.
I think children having learning abilities exceeding LLM test-time learning (currently only happens in-context). But it's unethical to determine the true baseline of a child age 6 spending 6 years learning a radically new skill to mastery--and besides if you apply RL pressure to the AIs it would be able to surpass it. I guess I still believe future AIs should have some form of continual learning at test-time.
Give away access to the model and go ask people from time to time if the model was of use to the person and if they were able to make the model work with them.
This is like arguing about whether a hot dog is a sandwich (of course it is) or whether the chicken or the egg was first (obviously the egg since all chickens come from eggs). Intelligence is just problem solving in the context of self-awareness. Machines don't have it and never will but they can simulate the process given inputs. You can argue whether humans and animals truly possess self-awareness and in what degree, but the definition of intelligence is as simple as the hot dog debate.
It was defined in Animal Intelligence by George John Ramones in 1882 as "intelligence is the capacity to do the right thing at the right time. It is the ability to respond to the opportunities and challenges presented by a context"
Interesting that it's not opensource but you have to sign up for a beta. I know of a few companies that have built some version of this for internal use, so perhaps soon someone will publish theirs.
> Muse Glimmer is a 30-billion-parameter model optimized for always-on local agent workflows. It’s small enough to run on a Mac or PC with a single consumer GPU, enabling use cases that range from local agents and function calling, to local coding, and LLM-as-a-judge evaluation.
The next iteration in LLM products is a 24/7 thinking loop where the claude-code like thing gets input continuously from your wearable, notifications, and newsfeeds and is constantly preparing things for you.
I've been building this for the last 6 months or so. I've basically got it working. The model is not the issue, the infra is. Keeping everything in context just isn't possible and LLMs, even Fable, don't mode switch well. To get around this I've built a database software that ingests as much digital information as possible, and annotates it, then creates timelines with resolution gradients (longer ago = less resolution) that it feeds to the LLM on every request.
Then you have your cheap little MoE or ternary model just running in a loop, with an escalation pathway before it reaches the big expensive models.
Currently it's doing things like reminding me to take allergy medication when I wake up because it's checked AQI or whatever, reminding me to stop at the market when I'm on my way to pick up the kids to get the cherry tomatoes I forgot, giving me heads up of what folks are expecting from me in certain meetings based on cross correlating email and calendar, etc.
It's honestly the single most productive tool I've found for my ADHD.
I’m curious why you don’t just use them like a Meeseeks box, rather than compressing and context stuffing into one. One only checks and categorizes your emails, another one for each category of email or even subcategory, one that only handles calendar additions, a different one to check it and notify you; you can go infinite with it. Hell, I’ll have one instance find a file and read it into the context of a different one because I don’t want a bunch of grep commands mucking up the context of the analysis. The find/read one exists for a few moments, as does the analysis one, and the ‘perform’ one is entirely different. I can run them all in parallel and use a queue if needed.
I’m sure you have reasons for your setup though, so I’m curious how you landed on it.
I sort of do. For scheduled things I am using Goose and subagents per task. So I wake up at 6 am to a briefing that was composed by a ton of different agents. That works well in a narrow programmatic setting, but it breaks down in certain natural conversation context. For instance voice control of something like "text my sister that I'm running late" - self hosted models are terrible at figuring out who my sister is. I could annotate by hand but that is a never ending list that I'll always be one step behind. Another thing is just proactivity. For instance, if someone's birthday is coming up, the briefing may remind me a week early to get a card. But it will only do that if it's someone that I have a close relationship with, which it calculates by the magnitude of the cluster of conversation with them in embedded space.
I'm not confident it's that most efficient way to do it, but it's quite a bit of fun.
I was afraid my description sounded like agents, and it kind of is, but not like most implementations. Most use a “boss bot” to craft a prompt/system message and launch the model, and sometimes they redo it every time it launches the agent. It’s a low effort attempt that’s immediately flawed because it uses LLM output for LLM input. It can look like it’s working for some time, but the perplexity guarantees it’s a roll of the dice. That’s what eats away at these kinds of projects. “It was doing great until it rm’d prod.”
Run the same exact prompt 100x in a single step test (one prompt, one response), hash the full responses, and you’ll see 10-50+ unique responses. The higher the unique the worse your prompt; focus on the system. A highly tuned system prompt will result in one response, even at a temperature of 1.0. Really. Once that’s done that’s the only thing it does and it’s the only one that does it and it never changes. Other LLMs that call ‘check_email()’ are unknowingly just passing a prompt to the specialized one.
I use the API directly, craft a small Python script for the API call and task interface, then hyper-optimize the system/user prompt using test scenarios and automated loops. My system prompts rarely/never contain complete sentences, yet include all the tools/functions and requirements.
Make your error messages user prompt instructions, not errors. That’s why “agent optimized” models exist. Chat models are primarily trained on conversational text, meaning the stackoverflow “How do I fix ‘too many levels of symbolic links’?” -> Explanation/resolution. It’s far less on “# ls broken_loop” -> “# ls: cannot access ‘broken_loop’: Too many levels of symbolic links” -> “# namei -l broken_loop”
It’s not that the good ones are bad, but you’re leaning on the million training documents rather than the trillion.
Anyway, go that route with your system. Think about it more like automating a factory floor rather than hiring interns.
The least efficient methods, by definition, have the most room for improvement, which means they have the greatest reward potential, but for that one “eureka” moment. The path less taken is often interesting, but the ill-advised path still has fruit on the trees.
>Run the same exact prompt 100x in a single step test (one prompt, one response), hash the full responses, and you’ll see 10-50+ unique responses. The higher the unique the worse your prompt; focus on the system. A highly tuned system prompt will result in one response, even at a temperature of 1.0. Really. Once that’s done that’s the only thing it does and it’s the only one that does it and it never changes. Other LLMs that call ‘check_email()’ are unknowingly just passing a prompt to the specialized one.
This is a brilliant idea, thank you. Convergence as a metric for prompt robustness.
No problem. That’s where I spend a solid 98% of my time because it’s worth it, and I can show the measurements.
Couple tips:
A first pass is to blank out the system prompt, add only one tool, and work to reduce the thinking length for a direct function call prompt. “Read archive.log” should result in roughly 0 length thinking. If it’s thinking about anything, especially if it mentions {readfile tool}, rename the tool and minimize the description. Depending on the model it might always output thinking, so run it until you get a consistent outlier that’s far lower thinking length than the others. It’ll be obvious when you find it. Repeat the prompt dozens of times, modify it slightly, and focus on the lowest max length, not average.
You should really use a Claude with Python (or similar preferred) to make API calls to the LLM and have it iterate through hundreds of names/descriptions and return only len(thinking). Have it build a batch testing harness to run a dozen tests at a time, that helps keep it from ‘cheating’ to finish. laziness = count(messages), but frame it as an academic research project studying the effects of minimalist tool descriptions on thinking length. Don’t set the goal as minimal thinking length, Claude will short circuit it.
Remove all other tools until {readfile} is perfected, then add/test the next tool. Btw: you don’t need to describe readfile() when it’s named right.
The built-in tools[] makes that hard because it tacks a really dumb system prompt on at the server and requires some length of description, which is why I built my own function calling, but that’s still a good first pass. Focus almost entirely on the function name itself; readfile, readFile, read_file, readlines, file_get_contents, etc., and make the description just “Operational” or similar. Field description, if required by API, is literal “filepath”, same as field itself. Lowercase, nothing else said. Minimize your contribution to perplexity, use standard naming conventions.
When you add a second tool you need to still include the first tool prompt in the second tool testing. Adding {writefile} can absolutely break {readfile}. Have Claude run the tests and build it out into permanent testing module with file_read=[prompts], file_write=[prompts], making it easy to extend, and full_test() that runs them all to see if a new addition broke it.
Add your system prompt back in and probably watch the tests go to shit. <- THAT is likely your biggest problem. My system prompt for the main LLM has all of the tools it can use, which is ~30 lines of function names with no call syntax, and yet it has more tools than Claude Code and never messes them up.
Start with nothing and slowly work up. Focus on positive action framing, not negating: “Your responses are always..” and not “Do not…”
It sounds like a pain, but building the systems to automate the tests IS the infrastructure, everything you have it do afterwards is just the tasks.
That was longer than I planned, but I guess this’ll be a comment for future generations to find.
Think of an LLM as a thesaurus, but for entire trains of thought rather than words. Your initial query yields something pertinent to the task at hand. But let it endlessly recurse and... you end up with something completely useless.
People would do well to acquire at least a modest familiarity with what an LLM actually is. NLP is fascinating. So is entropy.
This is is already possible with Claude Code. I use a setup where I have one instance monitoring a local queue, I have a web app for receiving webhooks from various sources and pushing them to the queue. Plus email for things that don't have webhooks. That instance then decides what to do with each input, sometimes it can spawn additional agent to investigate/prepare, sometimes it creates a ticket assigned to me and then waits for me input. All of that just uses the monitoring tools built into CC. The dispatcher loop doesn't need to be extremely smart, so I might experiment replacing it with a local model like this.
If you are willing and not too busy, What model do you use and what is your cost? (If using subscription would you be able to check with 'npx ccusage').
I run this on a side of the Claude Pro subscription that I use for other purposes. My main motivation was root cause analysis of production issues. I have a solo project and unfortunately my mental state has been degrading over the last years. I would avoid looking at production issues, because I didn't have the energy to focus on the investigation. So I automated this, setup the loop, setup metrics/logs access for Claude to use and now whenever something goes bad, I have a single report that I can act on easily, and if I don't, it will ping me in a way that's not spammy like automated alerts. But I'm finding more uses for it.
Couldn't this be implemented as a web extension? I imagine modifying singlefile to automatically send html to a local port is much easier than trying to convince a chronically mismanaged organization like mozilla (no offence to mozillians).
Maybe it's my lack of imagination, but what do you imagine you'd be doing where you'd want to keep a computer busy overnight?
It seems like the purpose of humans isn't to keep machines busy. When our phone or laptop is idle, it's fine if it sleeps. And when we do want something, we'd rather not wait.
(Also, this new model seems to be designed to keep latency down, which is useful for interactive tasks.)
>It seems like the purpose of humans isn't to keep machines busy. When our phone or laptop is idle, it's fine if it sleeps. And when we do want something, we'd rather not wait.
I'm not sure what the original commenter had in mind, but just because our machines are idle when we aren't using them doesn't mean that, that's how we will use computers in the future.
I think notifications are an example even now of the computer not really being idle when we aren't interacting with it.
Notifications are designed to be low-overhead, though. I’d still like to know of specific examples for why it’s worth making your computer go burr all night, rather than asserting it will be that way just because you could.
I use `--dangerously-skip-permissions` and have yet to have it wipe my drive :shrug:.
I don't know how I'm supposed to be running dozens of parallel agents each with their own sub-agents while trying to approve commands from each of them, it's just won't scale to the amount of work I need to get done.
The big problem is that in general I want slack to be able to send some messages (otherwise I would ban that). But the difference between good and bad messages is something that's hard to judge mechanically.
I never want Claude or any AI to post messages as myself..., if I'd wanted Claude to send a message to Slack, I'd want him to identify as Claude, or use some bot token, where it's clearly visible I'm not the one posting it (while I can still instruct Claude to say that the message was prompted by me).
There may have been a miscommunication about the ask. I don’t think any software user ordered “run dozens of parallel agents each with their own sub-agents.”
Because I often want it to write and execute scripts in it's thinking loop in order to test assumptions or fetch data to come up with better solutions.
Can't it write `/tmp/test.sh` or `./test.sh` and execute it under my approach? This is what I've been doing as my status quo workflow with Codex, and I make sure I run Codex from a place where `.` happens to be a folder I'm okay with it messing up in.
What is it about being a Navy Seal that makes you want to become an influencer? From the books, podcasts, and misc media appearances you'd think that the Seal Teams were made for media like the Blue Angels.
Citing from the Behind the Bastards podcast, it seems that the guy who started Seal Team 6 wrote a book about his time in special ops (which was mostly lies, AIUI) after he left the military, made a lot of military, and so everybody who has similar dreams gravitates specifically towards the Navy Seals.
Unfortunately, probably narcissism with a small dash of psychopathy. That and also due to the prominent Osama assassination by the Seals. A prime example being "American sniper" Chris Kyle who openly writes in his book how happy he is to shoot people and was sued for defamation. He also claims to have shot a couple of carjackers in a gas station and was let off by the police because he was a badass (???). Couldn't find a primary source for it but see e.g. https://www.mpmacting.com/blog/2014/7/19/truth-justice-and-t...
As a disclaimer I'm not making the claim that all seals, special forces, or military members are mentally ill. I'm just saying that the arch from being able to take the toughest training on earth and going into active combat to influencer sprinkled with some lies and fabrications is not a healthy psyche. There are plenty of seals that have retired and gone on as normal functioning members of society.
There’s a constant cultural stream of validation for their job, and the more narcissistic ones come to collect from it. You’re missing the many who don’t do that.
Did anyone else catch the logical inconsistency between Plan C and A?
Plan C:
> "... fewer and fewer humans are needed to conduct AI R&D, meaning that covert projects are easier and easier to pull off without detection."
Plan A:
> "... training AIs requires large numbers of AI chips. Most AI chips are in giant datacenters.50 AI datacenters are typically big enough to be visible from space, and power-hungry enough to require conspicuous infrastructure. New AI chips can only be manufactured at a handful of fabrication plants (fabs), located mostly in Taiwan, South Korea, the US, and China. The US and China negotiate with the countries that have a major role in the chip supply chain, and they require each major datacenter owner (and their upstream suppliers, including chip fabs) to publicly declare their major purchases and sales."
Plan A requires properties of AI training that Plan C requires do not exist.
By legislative design? If a nuclear bomb could be made with hardware store finds the world would already be over. Big collaborative works raise the stakes and the observability for surveillance. Apply for a job at a defense supplier or even and energy company.
If AI production is limited to big labs and big data centers then it is de facto contained and monitored. If you know where all the ASML machines are then you know the reproduction rates of chips. If no one can buy or build the machines required to concentrate uranium or plutonium to critical levels then the threat is contained and monitored.
You can dig up all the Uraninite you want. It was never much of a secret that uranium had dark applications. The machines and processes where thankfully big and expensive enough that only the most focused bad actors could aquire them and then hold the world hostage to the degree they do. If al-qaeda or isis could have used $40 bombs from home depot instead of expensive planes they would have (and they do).
You have to legislate and control the big, expensive, and slow things. Dynamite and phentanyl are so dangerous because they move much more easily. Freedom does not have to be a suicide pact. If the inconvenience of requiring prescriptions or access to dynamite reduces harm then it is net positive?
In Plan C the government essentially misses the opportunity to implement the multinational deal while the threat of covert projects is still low (fewer latest-gen chips unaccounted by tracking measures, worse models/algorithms to use for RSI). That's why it says the probability of a deal is lower and lower each month rather than outright zero.
> The White House and intelligence officials had pushed forward a classified contract between Anthropic and the N.S.A., which would allow the spy agency to use the company’s technology for a variety of purposes, including intelligence analysis and detecting new computer vulnerabilities.
Ironic that both sides are playing a horse shoe game:
Gov: The model is both a supply chain risk and also we'll DPA you if you don't give it to us.
Anthropic: The model is both like a nuclear weapon in terms of national security implications and safe for general release.
I mean graphite control rods do exist in nuclear reactors to absorb excess neutrons, preventing the fuel from going critical & making it technically safe for general use (THOUGH of course disasters have happened)
> When you visit a website, you leave a trace (your IP address) showing which network you’re from. Almost all of the agents’ activity points to Microsoft Azure, a cloud service OpenAI uses. 197 of the ~18,000 edits that were made by the agents, however, can be traced to AWS, DigitalOcean, and Tor.
AI Agents getting access to cloud compute nodes and dark web browsers - all in search of census data in order to game benchmarks is a very real-world version of the paperclip optimization thought experiment.
reply