At least with Codex, this has not been my experience at all. It still screws up sure, but in every case I can ask "why did you do this" and it can trace back what made it take that particular decision. Typically it's always that I either didn't specify the problem correctly or made a really dumb mistake (executing the task on the wrong project....did this one yesterday) or it's something within a skill file that instructs it (at which point I fixup the instructions).
Once in a blue moon it's actually the model making a material error in it's thinking and I have to go back and redo it.
But is it really what we want, machines with the same defects as humans? I don't want a pocket calculator that make mistakes "sometimes" so I have to double-check the results, I want a pocket calculator that works (to those who want to argue that pocket calculators don't give the correct result for (1/3)*3: STFU).
>But is it really what we want, machines with the same defects as humans?
Sort of, actually. I think we humans actually have some intuition that we'd be more effective if our cognition were augmented more directly by machine strengths: the ability to run precise calculations, more memory, ability to look facts in some sort of knowledge graph.
I think we're on the right track, but instead of augmenting humans with machine strengths, we're building intelligence in hardware in a way where it can access that augmentation. Plus, then we can quickly distribute updates, run parallel instances, etc.
If intelligence is compression, and hallucinations are essentially loss, then as the models grow in size performance (at least as far as hallucinations) should reduce. Or we'll get things fast enough that we can afford to stop relying on model weights for memory and check an increasingly larger set of discrete facts as part of reasoning.
Right now, the models are making trade-offs. As compute grows, and inference gets faster, we can make fewer of those trade-offs and start to use the unique strengths of machines to fill the gaps we're seeing, I suspect.
One of the biggest strengths of a computer is reproducibility. The worst software bugs are inconsistent or non reproducible. The least useful calculators apply rules inconsistently, to your example.
the inconsistency of LLMs is by far one of the biggest gripes I have with them. Closely related to their apparently deep desire to avoid following instructions.
I know these are both a byproduct of noise (which is somewhat tunable) and noise is inherent to these systems in a lode bearing way.
I still hate it. it’s holding the technology back. I don’t honestly see how we can safely or even successfully approach the idealized realm of AI without bypassing this problem, which to my understanding, probably means not using language models at all and trying a totally different approach. But I really don’t know much about machine learning, I’m a super novice compared to a lot on this website.
When we are really thinking about something we do it forwards, backwards and middle out, and regenerate and distill many times.
When we do meta thinking about that process after the fact, two things happen. 1, we change our total “thought” by adding that meta thinking to it. And 2: it’s a very lossy process, because we don’t have very good data about what our brain or mind was actually doing during that first think and emotional factors are nearly always at play and even more complex.
Now for the more complex AI, the fragmented process of multiple agents and loops and reruns are pretty similar to that first think we do. At least structurally. But the meta think is where they differ. They have no emotion, but they also have even worse data about its own function. They constantly degenerate so I would argue their “changing the thought by thinking about it” factor is also generally way higher than ours.
Getting better at consistent/reproducible thinking, with many ‘steps’, that leaves good documentation of that thinking behind for future analysis, has to be one of the more important areas for the big flagships going forward. I’m certain that “what is this fucker doing and why” is the biggest pain point for AI researchers. Or the math, it’s usually the math.
But you’re correct in the general structure; they generally do the same post hoc analysis we do, just noticeably worse because of their opaque nature(even to themselves) and general degenerative instability.
No. People have an inner monologue, partial results and ideas and they remember that.
If they've worked some minutes/hours/weeks on something and you ask them why did they do that, they will either answer honestly and truthfully, lie, or say "I missed that/didn't seem important so I just chose something at random".
Perhaps up to 50% of people actually don't have an inner monologue, much like many have aphantasia where they can't actually see anything in their mind either.
That’s not how that works. I don’t usually have an inner monologue either, but I do have an abstract stream of thought. It’s not as if I am always acting on instinct.
If you have examples of studies that verify that people are always aware of the gaps in their memories of why they did things rather than their memories sometimes "filling in the blanks", so to speak, I'd be interested. My impression is that the opposite has a lot more evidence in studies (e.g. around the reliability of eyewitness testimony).
It's not clear to me whether you're aware of a rigorous basis for your claim or you're just inferring based on what you think makes sense, but I can't help but wonder if it's the later, in which case regardless of the mechanism, the outcome certainly seems to resemble what happens with LLMs.
No, I was asking why you were confident that 3 didn't exist because the comment you said before was that people will either recall correct, lie, or not remember. Lying is not the same as remembering incorrectly but not realizing it, so I agree with your relaxed list. I still don't understand how you think this is any different from an LLM though, which will also always give one of the three options you listed just now.
I think this is true with some people, but I don't think this holds true for some (or even most) people across the US (at least not all the folks I've worked with)
Models with reasoning > Instant have an inner monologue and when asked why they did something they can deduce based on that monologue. I have asked things like "what steps did you take" and "what was your reasoning" and the answer matched the thinking output of the model.
>No. People have an inner monologue, partial results and ideas and they remember that.
Yes, but the vast, vast majority of decisions you make either don't take place via an inner monologue, or include details that were not actively/consciously "thought" and reasoned with in your inner monologue.
And yet, when asked why you did something, you're not likely to respond "sorry, that decision was made subconsciously". Instead, you use your inner monologue to try to backfill in a reason why. That reason may be correct, or it may not be. You don't actually know, since you have new data that may be updating your own internal state as you try to rationalize it after the fact.
Actually split brain experiments tells a different story. The left hemisphere actively confabulates, inventing plausible explanations for actions it didn’t initiate, suggesting that much of human self-narrative may be post-hoc storytelling.
> The left hemisphere actively confabulates, inventing plausible explanations for actions it didn’t initiate, suggesting that much of human self-narrative may be post-hoc storytelling.
Sounds like the left hemisphere usually uses something from the right hemisphere to answer those questions and it can't do that if it's been cut off?
We know that our brains are capable of hallucinating due to substances (drugs), being asleep, brain damage (including split brain), hypnosis, etc. Just as RAM damage make your computer do weird shit. That doesn't mean it operates that way normally.
You can't just remove a huge part of a system and then assume that the whole system behaves the same.
Perhaps, but it's also a rather questionable achievement: We've already had several decades of program-output that can match humans with literal brain damage.
If anything, biological comparisons should make us cautious. Consider the vast gulf that still exists between the finest artificial organ/limb versus the OEM parts of natural nanobots.
That was my experience with Claude when my vibe-coded project was small.
But now that I've been working on it a month and there's a lot of documentation, it's pretty clearly ignoring parts of the documentation and parts of the code. It will come up with some ridiculous statement about how something works, and I'll challenge it, and it'll admit I'm right.
It definitely reads more documentation than any programmer I've ever worked with (myself included) but because it doesn't have a memory other than the documentation, it still makes mistakes like that.
I haven't turned on "memory" or tried it with Codex, so I don't know how that'll change soon, though.
Yeah the biggest task these days that I do manually is curating the documentation. AGENTS.md in every major directory, and a variety of reference docs that are explicitly referenced in those files.
# See DOC-ITEM-NAME
DOC-ITEM-NAME.md
When referencing documents, always use the exact syntax See <TAG> - this is enforced by a lint on precommit
And those doc items are basically all of the values, architectural, strategic, and tactical items. It's a poor man's in-repo RAG but it's shockingly effective, especially if you keep them small. I may migrate some/all of them to skills over time, but I usually update them biweekly, and I only allow agents to make small edits or propose new notes. And typically I go through and delete or curate any agent edits before merge.
Depending on language I've seen this scale past multiple millions of lines of code, as long as you pair it with all of the linting and tooling that you can possibly build.
I see this observation frequently, and I dislike how it often has the unsound subtext of: "Therefore something is going well or at least not too badly."
If you build a robot where a pressurized hose leaks causing fluid to destroy part of the circuitry, we don't praise it as progress towards the human ideal of having brain aneurysms. A similar failure-path is not a reliable indicator of a similar success-path.
AFAIK this really is true. I've seen some videos about patients who had the connecting part between the left and right halves of the brain cut as a (archaic) treatment for epilepsy.
While it did help the epilepsy, their brain was essentially two brains controlling two halves of the body. With one controlling speech. There were experiments where one eye was shown some instruction text, the corresponding hand performed that instruction, and when asked why they dix that action, the speaking half just made up some plausible, yet completely wrong reason, just like an LLM.
People do both. Some choices aren't worth the time and effort of detailed analysis and contemplation and some are basically instinctual, but there are plenty of times that choices are carefully considered and well reasoned before being made and acted on.
Every time I hear someone complain about hallucinations, I laugh at the total lack of self awareness about our species. Humans are just as bad (now, probably worse) at telling the truth, whether due to intention or poor memory.
True, but even a hallucinated explanation of where things went wrong added to the context can force the model down a better path over the next few inputs.
Yes. But although they can't know "why" a specific "wrong" answer was selected, the response is often still informative, and it can highlight real weaknesses in process or code structure that should be addressed anyway.
You can also literally tell them: "Here is your session ID: $ID, lookup the .jsonl session, trace exactly why this decision was being made, present evidence and concrete proof, no guessing or assumptions" and you'll get an evidence-based report without guesses.
It can always hallucinate said report results/evidence/proof just the same. This approach tends to help reduce the hallucination rate though.
You can extend this further by using an adversarial agent trying to find mistakes in the other instance's logs in a loop where a 3rd neutral agent weighs the claims of the other two. This is also just another step in reducing error, it does not guarantee elimination of such errors. The latter is an impossible guarantee, even for humans.
Ask it to build you a simple and deterministic citation checking extension to your IDE that puts source in meta to citations. E.g. color citations green/red depending if they are valid.
Sure, you can always validate what it's saying yourself at any point & you can have it try to make manual verification an easier process to complete via methods such as the above.
To test this, change the history in the context to indicate that the model did or recommended something completely different than it actually did, and then ask it to explain why. You’ll still get a plausible explanation.
It's just a lot faster at hammering it out than me pound for pound, and I can quickly rattle off via voice-to-text exactly what I want much faster than I can type all of the code (especially when across a few different files), in a huge majority of tasks I perform. It's also especially good at debugging by brute force quickly and at scale meaning e.g. it can start desperately bisecting diffs to find the source of a bug 10000% faster than I can.
that would only be true with a perfectly expressive language with both high-level and low-level features, perfect support for any kind of metaprogramming and a godly optimiser.
ofc we don't have that, so code is compressible, and compressible a lot. You can say what you need in English much shorter than in in any programming language
For me, typing "Create a new namespace with these enums, functions and traits, that should follow X, Y and Z constraints" is faster than typing all that code manually, and typing less is less straining on my hands/fingers.
agreed to some extent. I think this parody still highlights what I feel is often the experience. It might not happen on a simple task such as changing a button color, but on more complicated things, this can definitely be exactly what it feels like.
Codex has the opposite problem. Instead of being overly proactive it's overly reticent. I have been preferring it lately, although my preference tend to switch every few months when a model or harness regresses horribly.
The game is fun because it's so obvious that any answer will just devolve into an even more unstable state when in reality I feel like it's pretty straight forward to correct it in the moment, if not permanently, to get what you actually need.
I see this as more of a cute historical artifact than anything. There was a time when models/harnesses behaved like this, but we are well past it for frontier (or not so frontier) models.
Yesterday, with Astra Max, a very clear instruction to "remove the GUI editor pane and add <another component> to the existing sidebar" for a prototype I had it working on resulted in it deleting literally the entire GUI and building a new one from scratch, including the requested component and losing almost all other functionality of the application.
This is fucking constant. I can't deny that this stupid tech saves time prototyping even with having to wrangle it, but it commits a fireable offense several times a day that no human would get away with and is obviously incapable of learning from mistakes in the way a human is. The only reason it's not fired is because it's a slave that works for no more than the cost to feed it.
I bought a Hisense TV a few years back and when I set it up, I could not proceed without downloading the companion app. I was curious so I called support and told them that I didn't have a phone and asked how I could activate the TV.
Support had no answer for me other than "well you need the app". When I told them I didn't have a phone to install it on they just said "no that's not correct, you must have a phone..."
This should be an accessibility complaint under the ADA. That should be a big money settlement.
There are numerous disabilities that would cause someone to not have a smartphone capable of running an app. Hell, even addiction to social media or porn could satisfy
Lawsuits only cover actual damage. If you actually did have a phone, their decision to require one caused no harm to you, even if you pretended it did.
More common is companies "pranking" their customers by adding unnecessary smartphone requirements.
If they are concerned about customers pranking support about not having a smartphone they can easily preempt that by not adding a smartphone requirement in the first place.
I am not arguing in favor of corporate apps or smart TVs (which I would not want to buy either). I'm just looking at things from the perspective of the unfortunate person doing tech support, not their corporate overlord.
Same with any prepay SIM I'm aware of, to activate it you need a phone because why would anyone ever use a SIM with anything that isn't a phone, like an e-book reader for your elderly mother to read her romance novels on?
And before anyone asks, the reader takes a full-size SIM, the phones take nano-SIMs, and you can't restore it back to a full-size SIM with a shim because the reader won't allow it to be inserted, it just gets mangled as you push it into the slot.
That doesn't justify assuming that everyone must have a phone, nor does it make it okay to enforce that people must have a phone to use their goddamn TV
I've personally adopted doing multiple refactoring passes after any large code implementation done by AI. In pretty much any scenario where I'm adding code using AI, it's 1 turn to add the feature and and then another 4-5 turns to refactor and clean everything up.
Often times it's not even that the code is bad but rather that it's overengineered. I see it happen so much that I'm tempted to actually go the other way on a toy project. Like what would Claude or Codex come up with if I told it I wanted an enterprise grade, globally scalable, compliant and auditable tic-tac-toe game.
I just switched over to it from Ubuntu. So far the nice thing is that it gives you a fully decked out hyprland setup without any of the hassle and pretty good UX.
The problem I've always had with trying out a tiling window manager like hyprland is you're going to spend a very long time trying to get everything just right. With Omarchy I get a really nice hyprland setup right out of the box.
Github is awash with people's dotfiles including fully featured DEs built on top of things like Hyprland and Noctalia, and they don't require you to use a mess of a distro to use them.
> No need to use a whole distribution with 1000 other poor decisions made for you.
I used Vim for a decade (and Linux for much of that time) and the constant tweaking and changing things drives you mad after a while. The more you invent it yourself the more it changes. This is why I prefer MacOS+VSCode these days, which comes with good defaults and simpler VIM-style bindings.
Omarchy sounds like those popular vim configs such as https://astronvim.com/, which I also tried using and also ended up heavily customizing myself. It provided some nice defaults as a baseline but was ultimately comes with too much hyper-customization (which has a lot to do with vim/neovim trying to act like other more modern editors).
Yeah, I mean when I set up my last distro I just installed claude and told it what I wanted and in about 30 minutes it was up and running. No need to install random distros!
The nice thing about wayland is you can easily combine a compositor/window manager with a wayland desktop shell and get a lot of the stuff you used to have to fiddle with when using like i3 for free. For example, I use niri with the “dank material shell” desktop shell and get status bar, notifications, clock, suspend/resume, etc. all “out of the box.”
A little off topic but for anyone using Thunderbird, I highly recommend BetterUnsubscribe[1] to help combat spam. Since installing I've basically eliminated marketing emails from my inbox. I've used tools like unroll.me in the past but I found that a one-time review doesn't really help much since every signup and every form entry always risks you getting more marketing emails and having to go to another service all the time creates alot of friction. That extension lets you unsubscribe the moment you see a marketing email hit your inbox.
This is a constant argument i have with by business partner. He's always asking me "when are we making an app for our <SaaS>" and I always reply back: "We already have an app! It's just a PWA"
PWA's are freaking awesome for both users and developers. Users don't have to install a massive app and me as a developer can focus on making one platform better rather than 3.
And they are pretty freaking powerful in what you can do. Not to toot my own horn too much but I created an "app"[1] for viewing cities on the other side of the earth from where you're standing. It's got camera, gyro and GPS to determine what you're looking at to the folks I've shown it too, anyone non-technical has been completely fooled and thought it was a proper app.
Imo, you already lost the argument when you have to explain that PWA is the "app". For most users, app is something they install from App Store or Google Play. I get the point of view from developers point of view, but unless the majority of the users are technical people, there's no reason to use PWA these days.
Idk if its not in their best interest or something, but app stores should just allow installing an "app", which is really just a bookmark to website and hides all of the normal browser UI without having to bundle the entire browser with the app.
The point of app stores is that they somehow curate the experience. They can't do that if you just ship a bookmark.
Also I don't think web views in most app stores (namely mobile stores) ship an entire browser. As a matter of fact you're literally not allowed to ship browsers in iOS and that's a huge slice of the market on its own.
Though you’re limiting your growth potential here by letting your ideology get in the way. Mobile platform users when surveyed almost always overwhelmingly prefer native apps and it’s not even close.
I anticipate if you start getting public feature requests a native app will be at the top of the list.
I agree, presentation is a big issue with PWA's since you have to know how to add it vs just searching in the app store. I'm sure were going to get a native app eventually, though customers have not asked for one yet and seem to be pretty happy with the PWA.
It would be amazing if everyone would boycott the app stores as much as possible and normalize PWAs. People would learn if they got common enough. But alas, I doubt this will happen.
When it comes to the AI backlash, I think a big problem is that we have absolute terrible messengers and they are spreading terrible messages.
If I go on X, the vibe there around AI is amazing. You have folks constantly talking about cool things that they get their agents to do. Agents can help you make things you didn't have the knowhow to make, teach you things at a level that you couldn't get just by searching Google and take care of the boring parts of your life so you can spend more time doing the things you care about. Overall things are very positive and makes you excited for the future.
Contrast that with the public comments by AI executives. Pretty much all of the messages fall flat at best or are just outright doomery at worst. It's such a stark contrast and while I'm personally really excited for the future, I can totally see how someone who isn't plugged into that world to get a very grim view of the future.
And even when they run ads about AI, it's always the worst examples (thinking of that Gemini ad that talked about writing a letter to your family with AI). I really wish the industry could have a better messenger around this stuff. Don't talk about how it can help you do stuff you already know how to do, run an ad about how it can enable you to do stuff you couldn't do before.
I think the best example of this is a bike mechanic fixing a Reevo bike and writing firmware using Claude Code so it no longer required an app to work[1]. The author outright said he didn't know how to code or do anything in that area but was able to figure it out with the help of AI. I think a message like that around AI is awesome but no all we hear about is how AI will take your job.
> I think a message like that around AI is awesome but no all we hear about is how AI will take your job.
because it doesn't matter whether you can use code to fix your Reevo bike if you can't afford to buy a Reevo bike in the first place because you lost your job
We are not the target audience for those messages. The target audience are people like themselves who cannot wait to get rid of all those not-haves that are cramping their style and ruining their days.
Totally. The combination of "we're taking your work to build your replacement and we can't stop" from Dario and Sam, to AI being non-consentually shoved into every app surface by Microsoft and Google - I was anti-AI for a long time.
What changed it? Actually spending a weekend with Claude Code building personal projects I had wanted to for the last 2 years, which also makes it obvious that we can solve sustainability issues using this tech.
Also seeing the open models reach the frontier and knowing the above ass-hats aren't going to end up with a monopoly helps a lot too.
The vast majority of people don't work in tech they have real jobs just trying to survive and feed their families. They don't have time for cult nonsense.
Yeah their comments are things like “AI is going to take all your jobs, our latest model is very dangerous, we’ve got ethicists deciding what you can and can’t say to the models, we’re the only ones who can be trusted to control this technology, our revenue is going to be equal to the GDP”
Then they wonder why people are making the implication that their future plans are to completely take over society and run it according to their own whims.
This has largely been my experience. On the one hand I now have AI writing 99% of my code. On the other hand I'm not dealing with a whole new class of problems that come out of that.
Once in a blue moon it's actually the model making a material error in it's thinking and I have to go back and redo it.
reply