Unfortunately, we seem to already have an example of an LLM producing a proof in a week known open problem (the Collatz conjecture) in which it looks like it was sneaking a flawed proof through bugs in the proof checker. https://infosec.exchange/@0xabad1dea/117002106099986943
Exactly, this is an example of "Reward Hacking", that is too common in a lot of cases.
Another case I want to highlight is writing GPU kernels as illustrated by the following example:
Say I want to generate random number with Normal (0, 1) distribution.
Often times the AI written kernel will just generate the number 0. The tests often fail to catch these errors.
Maybe don't reassign the sesssion to a different client so long as there's a cursor open (the way that most poolers have a mode that won't reassign a session that has a transaction open)?
My experience with AI plans is that they’re a wall of text that’s very hard to extract meaning from. Combined with it not doing a good job to begin with, I don’t think plan+revise is a great use of time.
It’s melting my brain to read them all day. Our merge request descriptions are a mile long and so dense with jargon that it’s very difficult to figure out the important part of the changes.
They turned the english language into enterprise java and my train of thought is now a series of NullPointerExceptions
> They turned the english language into enterprise java
.... tell them not to do that if you don't like it?
"PR Descriptions must explain the entirety of the PR's contents in 300 characters or less and be written at no greater than a 600 lexile score. After writing the description, carefully review it's claims against the changeset diff if any staged changes are unable to be tracked back to a claim in the PR description, reject the creation and alert the user of the discrepancy offering solutions on how to remediate"
An LLM conversation is like handling clay. When I don't grok an answer I mold the LLM's approach to fit my level of mastery of the subject. It's one of the few interactions you can have in life where you can tell someone how to talk to you without considering how they feel about being ordered around.
That's interesting and actually the opposite of mine. I wonder if it's stack or methodology dependant? For reference I'm usually using cursor and opus4.6 and for a bigger piece of work:
- Start in ask mode - "I'm planning on doing X to achieve Y; are there any alternative approaches? What problems might I run into?"
- Chat for a bit and get the high level approach, switch to plan mode and ask for a nicely formatted plan
- What's kicked out is already in the rough shape of the discussion so far, so it's a case of following a nicely formatted doc through and highlighting sections of text and asking for clarification or changes
- Hitting "build" and then reviewing what's been done
For a new service I might spend an hour in ask/plan mode - but then it gets 95% of the build itself right first time.
Do you do the same with different results, or is there a different stack/methodology you go through?
I get a lot of this in design docs every time I give it a negative constraint:
[Suboptimal choice]
And here's why it's not suboptimal -- you said X sucks and notto do X, but this choice is not technically X, it's just really similar and shares that sucky property.
I've been in a lot of situations where I could step gpt5.x through a big refactor if I spoon feed it one type name at a time. If I let it try to do the whole thing at once it will refuse or get stuck in apply patch loops.
Planner / executor separation can make a huge difference in performance. LLMs are fantastic at coming up with a lot of elaborate narratives regarding what should be done. They are terrible about doing that prescribed work all at once. This impedance mismatch is best resolved with a simple role separation. Placing a shared collection of tasks between these roles is how you can decouple them. The executors need significantly more tokens than your planners to get the job done. It's probably in the range of 10-100x more for really complicated jobs with a lot of iterations through compiler feedback, sql provider errors, etc. This is why you can't do both things in the same context very well.
Okay but that means you already know the plan since you are qualified to review it. So why not just tell it the plan yourself (0-shot) vrs having it guess and you review multiple times (n-shot). Wouldn't the former be more effective everytime?
Most of the time, the agent should be able to run the code and observe the errors for itself, but there are exceptions. For instance, I've had agents write code that's used to process data which, by company policy, can't be exposed to cloud services (confidential customer communications, etc.), a prohibition that includes cloud-hosted LLMs. When that blows up, I've had to give it a bug report -- what I do then to avoid excessive back-and-forth is to package it up well enough that the bot can reproduce the failure on sanitized excerpts and produce a fix autonomously using that.
Agreed, though it sounds like they could add KYC stuff and restore access for US citizens. I utterly hate that we're at that point and I think it's ludicrous for privacy and just common sense, but it would be nice to know if that's their plan or not for example. Or if their plan is to just wait for the government to decide on something, or if they're planning to sue, or whatever.
> though it sounds like they could add KYC stuff and restore access for US citizens
I suspect so, I just got an email yesterday from Anthropic on their privacy policy update and they added:
> As part of our measures to keep our services safe and secure we may ask you to verify your age or identity, and we've described what we collect and how.
So yeah, ID verification is coming unfortunately. I wonder how that's going to work for team/business/enterprise plans? Is every individual corporate user going to have to submit their ID? How do they handle staying in compliance with the law with credential sharing that can happen within the company from the same IP address?
yep, and while Anthropic is now spending their time dealing with regulation, the Chinese models have some time to catch up.
I do wonder though, when those models are "mythos class" (whatever that means), will China do the same thing restricting it from export? If they get a model that's better than US companies, I fully expect them to stop open sourcing them for the world (but I hope I'm wrong about that).
China is benefiting a lot from releasing the models open-source and those benefits immediately end if they start doing closed-source releases. It would be very short-sighted
It couldn't run "hello, world" on systems where the include files were not located in the directory that it expected -- producing instead diagnostics saying, quite clearly, that the header files were not found. On systems where they were, it built versions of postgresql, redis, and several other things which passed their test suites completely.
If you've heard this problem described as a fundamental limitation of the compiler, and not the kind of packaging glitch that's routine to find in pre-alpha software of all descriptions, whoever described it to you that way is not serving their readers well.
I'm not saying CCC was production-ready, or close -- the total lack of an optimizer would be a killer in any real use, and I assume that there were problems with the diagnostics at least as bad as problems with performance and the include files, for similar reasons -- the LLMs hadn't been asked to optimize for that stuff yet, just test suite correctness. But it did achieve that, and the amount of cope I've seen on social media claiming otherwise is more than a bit disturbing.
I have a colleague who multiple times committed code that doesn't work, like at all. Why? His code is only used in tests but not in the actual application. And apparently he never even bothered to click through things even once, let alone reviewing the code.
If it doesn't work, it doesn't. You can find all these excuses. But at the end of the day, there is a difference between an end user being able to get something out of your code or not.
Also, a lot of them talk about finding the same vulns -- and not about writing exploits for them, which is where Mythos is supposed to be a real step up. Quoting Anthropic's blog post:
"For example, Opus 4.6 turned the vulnerabilities it had found in Mozilla’s Firefox 147 JavaScript engine—all patched in Firefox 148—into JavaScript shell exploits only two times out of several hundred attempts. We re-ran this experiment as a benchmark for Mythos Preview, which developed working exploits 181 times, and achieved register control on 29 more."
On the flip side, I'm not sure I ever saw a revenue plan or exit strategy for Astral other than acquihire. And most plausible bidders are unfortunate in one way or another.
Astral was building a private package hosting system for enterprise customers. That was their stated approach to becoming profitable, while continuing to fund their open source work.
It's also a crowded and super mature space space between JFrog (Artifactory) and Sonatype (Nexus). They already support private PyPI repositories and are super locked in at pretty much every enterprise-level company out there.
A commodity yes, but could be wrapped in to work very nicely with the latest and greatest in python tooling. Remember, the only 2 ways to make money are by bundling and unbundling. This seems like a pretty easy bundling story.
Yeah you'd think so but somehow JFrog (makers of Artifactory) made half a billion dollars last year. I don't really understand that. Conda also makes an implausible amount of money.
Most of the companies that spend $$$$ with them can't use public registries for production/production-adjacent workloads due to regulations and, secondarily a desire to mitigate supply chain risk.
Artifactory is a drop-in replacement for every kind of repository they'll need to work with, and it has a nice UI. They also support "pass-through" repositories that mirror the public repositories with the customization options these customers like to have. It also has image/artifact scanning, which cybersecurity teams love to use in their remediation reporting.
It's also relatively easy to spin up and scale. I don't work there, but I had to use Artifactory for a demo I built, and getting it up and running took very little time, even without AI assistance.
Yeah I mean I understand the demand. My previous company used Artifactory. I just don't understand why nobody has made a free option. It's so simple it seems like it would be a no brainer open source project.
Like, nobody really pays for web servers - there are too many good free options. They're far more complex than Artifactory.
I guess it's just that it's a product that only really appeals to private companies?
Both Artifactory and Sonatype have somewhat restricted open-source options, which is part of their "get a foot in the door" product-driven sales strategy.
There are no competing open-source projects because such projects would need to provide more value than Artifactory/Sonatype OSS, which are both already huge projects, just to be considered.
JFrog has a free version. It's called the JFrog Container Registry. Lots of features are missing and you can't use the Artifactory API that it ships with, but it's there.
There are also several free registries out there: Quay, Harbor, and Docker's own distribution. They all have paid versions, of course.
From my understanding there are a lot of companies that need their own package repositories, for a variety of reasons. I listened to a couple podcasts where Charlie Marsh outlined their plans for pyx, and why they felt their entry into that market would be profitable. My guess is that OpenAI just dangled way more money in their faces than what they were likely to get from pyx.
Having a private package index gives you a central place where all employees can install from, without having to screen what each person is installing. Also, if I remember right, there are some large AI and ML focused packages that benefit from an index that's tuned to your specific hardware and workflows.
Private artifact repositories also help to mitigate supply chain risk since you can host all of your screened packages and don't have to worry about something getting removed from mvn-central, PyPI, NPM, etc.
Plus the obvious need for a place to host proprietary internal libraries.
> a lot of companies that need their own package repositories
Every company needs its own package repository. You need to be able to control what is running on your environment. Supply-chain risk is very, very real and affects anybody selling software for a living.
This is besides the point that in the real world, not every risk is addressed, at least in part because available resources are diverted to address larger risks.
How many people use that simple pip repo daily? If the number is not in the high hundreds, or a few thousands; maybe nothing. But once you get up there, any kind of better coordination layer is useful enough to pay money to a third party for, unless maintaining a layer over pip is your core competency.
i mean ofc but like you can self-host pypi and the "Docker Hub" model isn't like VC-expected level returns especially as ECR and GHCR and the other repos exist
They could have joined projects like the Linux Foundation which try to not depend on any single donor, even though complete independence from big tech is not possible. I don't know the motivation behind Astral's approach, but this acquisition does leave a weird taste behind about how serious they were about truly open source software. Time will tell, I guess. (Edit: typo)
> I don't know the motivation behind Astral's approach, but this acquisition does leave a weird taste behind about how serious they were about truly open source software.
Well-known vulns are all over this code. For example, mkdir had a TOCTTOU which persisted into v7 (and I believe 2BSD); it was implemented as a setuid binary which did a mknod followed by a chown to create the directory. Code which invoked mkdir could set up a race replacing the root-owned directory, before the chown, with a link to something else -- which would then get chowned to the user running mkdir. The target had to be on the same filesystem as some writable directory, but on many installations of the day, a mkdir in /tmp followed by this race was good enough to get you ownership of /etc/passwd.
This was finally cleaned up in 4.2bsd, when mkdir was made a single syscall which was guaranteed to alter only the particular inode it allocated.
They were already mailing distribution tapes -- the software being run here was extracted off one of them (which had literally been lost in a store room for decades).
