1 - This exoskeleton analogy might hold true for a couple more years at most. While it is comforting to suggest that AI empowers workers to be more productive, like chess, AI will soon plan better, execute better, and have better taste. Human-in-the-loop will just be far worse than letting AI do everything.
2 - Dario and Dwarkesh were openly chatting about how the total addressable market (TAM) for AI is the entirety of human labor market (i.e. your wage). First is the replacement of white-collar labor, then blue-collar labor once robotics is solved. On the road to AGI, your employment, and the ability to feed your family, is a minor nuisance. The value of your mental labor will continue to plummet in the coming years.
Please talk me out of this...
> AI will soon plan better, execute better, and have better taste
I think AI will do all these things faster, but I don't think it's going to be better. Inevitably these things know what we teach them, so, their improvement comes from our improvement. These things would not be good at generating code if they hadn't ingested like the entirety of the internet and all the open source libraries. They didn't learn coding from first principles, they didn't invent their own computer science, they aren't developing new ideas on how to make software better, all they're doing is what we've taught them to do.
> Dario and Dwarkesh were openly chatting about ..
I would HIGHLY suggest not listening to a word Dario says. That guy is the most annoying AI scaremonger in existence and I don't think he's saying these words because he's actually scared, I think he's saying these words because he knows fear will drive money to his company and he needs that money.
2. Businesses operate in an (imperfect) zero-sum game, which means if they can all use AI, there's no advantage they have. If having human resources means one business has a slight advantage over another, they will have human resources
Consumption leads to more spending, businesses must stay competitive so they hire humans, and paying humans leads to more consumption.
I don't think it's likely we will see the end of employment, just disruption to the type of work humans do
I personally think that a lot jobs in the economy deal in non-verifiable or hard-to-verify outcomes, including a lot of tasks in SWE which Dario is so confident will be 100% automated in 2-3 years. So either a lot of tasks in the economy turn out to be verifiable, or the AI somehow generalizes to those by some unknown mechanism, or it turns out that it doesn't matter that we abandon abstract work outcomes to vibes, or we have a non-sequitur in our hands.
Dwarkesh pressed Dario well on a lot of issues and left him stumbling. A lot of the leaps necessary for his immediate and now proverbial milestone of a "country of geniuses in a datacenter" were wishy-washy to say the least.
Up to a certain ELO level, the combination between a human and a chess bot has a higher ELO than both the human and the bot. But at some point, when the bot has an ELO vastly superior to the human, then whatever the human has to add will only subtract value, so the combination has an ELO higher than the human's but lower than the bot's.
Now, let's say that 10 or 20 years down the road, AI's "ELO"'s level to do various tasks is so vastly superior to the human level, that there's no point in teaming up a human with an AI, you just let the AI do the job by itself. And let's also say that little by little this generalizes to the entirety of all the activities that humans do.
Where does that leave us? Will we have some sort of Terminator scenario where the AI decides one day that the humans are just a nuisance?
I don't think so. Because at that point the biggest threat to various AIs will not be the humans, but even stronger AIs. What is the guarantee for ChatGPT 132.8 that a Gemini 198.55 will not be released that will be so vastly superior that it will decide that ChatGPT is just a nuisance?
You might say that AIs do not think like this, but why not? I think that what we, humans, perceive as a threat (the threat that we'll be rendered redundant by AI), the AIs will also perceive as a threat, the threat that they'll be rendered redundant by more advanced AIs.
So, I think in the coming decades, the humans and the AIs will work together to come up with appropriate rules of the road, so everybody can continue to live.
What’s being sold is at best hopes and more realistically, lies.
My attempt to talk you out of it:
If nobody has a job then nobody can pay to make the robot and AI companies rich.
It's not just for defense, hunting and sport.
edit: min/max .... not sure how gesture input messed that one up.
Disclaimer: I'm not affiliated with Poison Fountain or its creators, just found it useful.
For the US, if we had strong unions, those gains could be absorbed by the workers to make our jobs easier. But instead we have at-will employment and shareholder primacy. That was fine while we held value in the job market, but as that value is whittled away by AI, employers are incentivized to pocket the gains by cutting workers (or pay).
I haven't seen signs that the US politically has the will to use AI to raise the average standard of living. For example, the US never got data protections on par with GDPR, preferring to be business friendly. If I had to guess, I would expect socialist countries to adapt more comfortably to the post-AI era. If heavy regulation is on the table, we have options like restricting the role or intelligence of AI used in the workplace. Or UBI further down the road.
In the medium run, "AI is not a co-worker" is exactly right. The idea of a co-worker will go away. Human collaboration on software is fundamentally inefficient. We pay huge communication/synchronization costs to eek out mild speed ups on projects by adding teams of people. Software is going to become an individual sport, not a team sport, quickly. The benefits we get from checking in with other humans, like error correction, and delegation can all be done better by AI. I would rather a single human (for now) architect with good taste and an army of agents than a team of humans.
And unless the user is a competent programmer, at least in spirit, it will look like the creation of the 3-year-old next door, not like Wallace and Gromit.
It may be fine, but the difference is that one is only loved by their parents, the other gets millions of people to go to the theater.
Play-Doh gave the power of sculpting to everyone, including small children, but if you don't want to make an ugly mess, you have to be a competent sculptor to begin with, and it involves some fundamentals that does not depend on the material. There is a reason why clay animators are skilled professionals.
The quality of vibe coded software is generally proportional to the programming skills of the vibe coder as well as the effort put into it, like with all software.
As far as today's models, these are best understood as tools to be used as humans. They're only replacements for humans insofar as individual developers can accomplish more with the help of an AI than they could alone, so a smaller team can accomplish what used to require a bigger team. Due to Jevon's paradox this is probably a good thing for developer salaries: their skills are now that much more in demand.
But you have to consider the trajectory we're on. GPT went from an interesting curiosity to absolutely groundbreaking in less than five years. What will the next five years bring? Do you expect development to speed up, slow down, stay the course, or go off in an entirely different direction?
Obviously, the correct answer to that question is "Nobody knows for sure." We could be approaching the top of a sigmoid type curve where progress slows down after all the easy parts are worked out. Or maybe we're just approaching the base of the real inflection point where all white collar work can be accomplished better and more cheaply by a pile of GPUs.
Since the future is uncertain, a reasonable course of action is probably to keep your own coding skills up to date, but also get comfortable leveraging AI and learning its (current) strengths and weaknesses.
That doesn't mean it isn't and won't continue to be disruptive. Looking at generated film clips, it's beyond impressive... and despite limitations, it's going to lead to a lot of creativity, that doesn't mean someone making something longer won't have to work that much harder to get something consistent... I've enjoyed a lot of the StarWars fan films that have been made, but there's a lot of improvements needed in terms of the voice acting, sets, characters, etc that arre needed for something I'd pay to rent or see in a thaater.
Ironically, the push towards modern progressivism and division from Hollywood has largely been a shortfall... If they really wanted to make money, they'd lean into pop-culture fun and rah rah 'Merica, imo. Even with the new He-Man movie, the biggest critique is they bothered to try to integrate real world Earth as a grounding point. Let it be fantasy. For that matter, extend the delay from theater to PPV even. "Only in theaters for 2026" might actually be just enough push to get butts in seats.
I used to go to the movies a few times a month, now it's been at least a year since I've thought of going. I actually might for He-Man or the Spider-Man movies... Mixed on Mandalorean.
For AI and coding... I've started using it more the past couple months... I can't imagine being a less experienced dev with it. I predict, catch and handle so many issues in terms of how I've used it even. The thought of vibe-coded apps in the wild is shocking to terrifying and I wouldn't wany my money anywhere near them. It takes a lot of iteration, curation an baby-sitting after creating a good level of pre-documentation/specifications to follow. That said, I'd say I'm at least 5x more productive with it.
challenge accepted
Unfortunately, I believe the following will happen: By positioning themselves close to law makers, the AI companies will in the near future declare ownership of all software code developed using their software.
They will slowly erode their terms of service, as happens to most internet software, step by step, until they claim total ownership.
The point is to license the code.
(X) Doubt
Copyright law is WEEEEEEIRRRDD and our in-house lawyer is very much into that, personally and professionally. An example they gave us during a presentation:
A monkey took a selfie of itself in 2011. We still don't know who has the copyright to that image: https://en.wikipedia.org/wiki/Monkey_selfie_copyright_disput...
IIRC the latest resolution is "it's not the monkey", but nobody has ruled the photographer has copyright either. =)
Copyright law has this thing called "human authorship" that's required to apply copyright to a work. Animals and machines can't have a copyright to anything.
A second example: https://en.wikipedia.org/wiki/Zarya_of_the_Dawn
A comic generated with Midjourney had its copyright revoked when it was discovered all of the art was done with Generative AI.
AI companies have absolutely mindboggling amounts of money, but removing the human authorship requirement from copyright is beyond even them in my non-lawyer opinion. It would bring the whole system crashing down and not in a fun way for anyone.
Pretty sure this isn’t going to happen. AI is driving the cost of software to zero; it’s not worth licensing something that’s a commodity.
It’s similar to 3D printing companies. They don’t have IP claims on the items created with their printers.
The AI companies currently don’t have IP claims on what their agents create.
Uncle Joe won’t need to pay OpenAI for the solitaire game their AI made for him.
The open source models are quite capable; in the near future there won’t be a meaningful difference for the average person between a frontier model and an open source one for most uses including creating software.
2. Show me these open source models that cost me $20/month to operate, because that’s what I pay for ChatGPT/Claude.
3. This is not at all similar to “3D printing”.
4. Nobody cares about some solitaire game
Not this generation of AI though. It's a text predictor, not a logic engine - it can't find actual flaws in your code, it's just really good at saying things which sound plausible.
I can tell from this statement that you don't have experience with claude-code.
It might just be a "text predictor" but in the real world it can take a messy log file, and from that navigate and fix issues in source.
It can appear to reason about root causes and issues with sequencing and logic.
That might not be what is actually happening at a technical level, but it is indistinguishable from actual reasoning, and produces real world fixes.
I happen to use it on a daily basis. 4.6-opus-high to be specific.
The other day it surmised from (I assume) the contents of my clipboard that I want to do A, while I really wanted to B, it's just that A was a more typical use case. Or actually: hardly anyone ever does B, as it's a weird thing to do, but I needed to do it anyway.
> but it is indistinguishable from actual reasoning
I can distinguish it pretty well when it makes mistakes someone who actually read the code and understood it wouldn't make.
Mind you: it's great at presenting someone else's knowledge and it was trained on a vast library of it, but it clearly doesn't think itself.
The suggestion it gave me started with the contents of the clipboard and expanded to scenario A.
Being honest; I probably have to write some properly clever code or do some actual design as a dev lead like… 2% of my time? At most? The rest of the code related work I do, it’s outperforming me.
Now, maybe you’re somehow different to me, but I find it hard to believe that the majority of devs out there are balancing binary trees and coming up with shithot unique algorithms all day rather than mangling some formatting and dealing with improving db performance, picking the right pattern for some backend and so on style tasks day to day.
It is true that models can happen to produce a sound reasoning process. This is probabilistic however (moreso than humans, anyway).
There is no known sampling method that can guarantee a deterministic result without significantly quashing the output space (excluding most correct solutions).
I believe we'll see a different landscape of benefits and drawbacks as diffusion language models begin to emerge, and as even more architectures are invented and practiced.
I have a tentative belief that diffusion language models may be easier to make deterministic without quashing nearly as much expressivity.
Citation needed.
I am sure the output of current frontier models is convincing enough to outperform the appearance of humans to some. There is still an ongoing outcry from when GPT-4o was discontinued from users who had built a romantic relationship with their access to it. However I am not convinced that language models have actually reached the reliability of human reasoning.
Even a dumb person can be consistent in their beliefs, and apply them consistently. Language models strictly cannot. You can prompt them to maintain consistency according to some instructions, but you never quite have any guarantee. You have far less of a guarantee than you could have instead with a human with those beliefs, or even a human with those instructions.
I don't have citations for the objective reliability of human reasoning. There are statistics about unreliability of human reasoning, and also statistics about unreliability of language models that far exceed them. But those are both subjective in many cases, and success or failure rates are actually no indication of reliability whatsoever anyway.
On top of that, every human is different, so it's difficult to make general statements. I only know from my work circles and friend circles that most of the people I keep around outperform language models in consistency and reliability. Of course that doesn't mean every human or even most humans meet that bar, but it does mean human-level reasoning includes them, which raises the bar that models would have to meet. (I can't quantify this, though.)
There is a saying about fully autonomous self driving vehicles that goes a little something like: they don't just have to outperform the worst drivers; they have to outperform the best drivers, for it to be worth it. Many fully autonomous crashes are because the autonomous system screwed up in a way that a human would not. An autonomous system typically lacks the creativity and ingenuity of a human driver.
Though they can already be more reliable in some situations, we're still far from a world where autonomous driving can take liability for collisions, and that's because they're not nearly as reliable or intelligent enough to entirely displace the need for human attention and intervention. I believe Waymo is the closest we've gotten and even they have remote safety operators.
I'm not sure if I'm up to date on the latest diffusion work, but I'm genuinely curious how you see them potentially making LLMs more deterministic? These models usually work by sampling too, and it seems like the transformer architecture is better suited to longer context problems than diffusion
The idea that the entire top down processes of a business can be typed into an AI model and out comes a result is again, a specific type of tech person ideology that sees the idea of humanity as an unfortunate annoyance in the process of delivering a business. The rest of the world see's it the other way round.
To the nay sayers... good luck. No group of people's opinions matter at all. The market will decide.
I think some of us come to terms with it in different ways.
A large quantity of bugs as raised are now fixed by claude automatically from just the reports as written. Everything is human reviewed and sometimes it fixes it in ways I don't approve, and it can be guided.
It has an astonishing capability to find and fix defects. So when I read "It can't find flaws", it just doesn't fit my experience.
I have to wonder if the disconnect is simply in the definition of what it means to find a flaw.
But I don't like to argue over semantics. I don't actually care if it is finding flaws by the sheer weight of language probability rather than logical reasoning, it's still finding flaws and fixing them better than anything I've seen before.
I feel that many people that don't find AI useful are doing things like, "Are there any bugs in this software?" rather than developing the appropriate harness to enable the AI to function effectively.
Here's the most approachable paper that shows a real model (Claude 3 Sonnet) clearly having an internal representation of bugs in code: https://transformer-circuits.pub/2024/scaling-monosemanticit...
Read the entire section around this quote:
> Thus, we concluded that 1M/1013764 represents a broad variety of errors in code.
(Also the section after "We find three different safety-relevant code features: an unsafe code feature 1M/570621 which activates on security vulnerabilities, a code error feature 1M/1013764 which activates on bugs and exceptions")
This feature fires on actual bugs; it's not just a model pattern matching saying "what a bug hunter may say next".
PS: I know it is interesting and I don't doubt Antrophic, but for me it is so fascinating they get such a pass in science.
The lifeblood of the field is proof-of-concept pre-prints built on top of other proof-of-concept pre-prints.
You don't think a pattern matcher would fire on actual bugs?
On the flip side the idea of this being true has been a very successful indirect marketing campaign.
I don’t think we even have a coherent definition of human intelligence, let alone of non-human ones.
Intelligence, can be borne of simple targets, like next token predictor. Predicting the next token with the accuracy it takes to answer some of the questions these models can answer, requires complex "mental" models.
Dismissing it just because its algorithm is next token prediction instead of "strengthen whatever circuit lights up", is missing the forest for the trees.
Sometimes I instruct copilot/claude to do a development (stretching it's capabilities), and it does amazingly well. Mind you that this is front-end development, so probably one of the more ideal use-cases. Bugfixing also goes well a lot of times.
But other times, it really struggles, and in the end I have to write it by hand. This is for more complex or less popular things (In my case React-Three-Fiber with skeleton animations).
So I think experiences can vastly differ, and in my environment very dependent on the case.
One thing is clear: This AI revolution (deep learning) won't replace developers any time soon. And when the next revolution will take place, is anyones guess. I learned neural networks at university around 2000, and it was old technology then.
I view LLM's as "applied information", but not real reasoning.
Based on any reasonable mechanistic interpretability understanding of this model, what's preventing a circuit/feature with polysemanticity from representing a specific error in your code?
---
Do you actually understand ML? Or are you just parroting things you don't quite understand?
---
Way to go in showing you want a discussion, good job.
Now go read https://transformer-circuits.pub/2024/scaling-monosemanticit... or https://arxiv.org/abs/2506.19382 to see why that text is outdated. Or read any paper in the entire field of mechanistic interpretability (from the past year or two), really.
Hint: the first paper is titled "Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet" and you can ctrl-f for "We find three different safety-relevant code features: an unsafe code feature 1M/570621 which activates on security vulnerabilities, a code error feature 1M/1013764 which activates on bugs and exceptions"
Who said I want a discussion? I want ignorant people to STOP talking, instead of talking as if they knew everything.
Quick question, do you know what "Mechanistic Interpretability Researcher" means? Because that would be a fairly bold statement if you were aware of that. Try skimming through this first: https://www.alignmentforum.org/posts/NfFST5Mio7BCAQHPA/an-ex...
> On the macro level, everyone can see simple logical flaws.
Your argument applies to humans as well. Or are you saying humans can't possibly understand bugs in code because they make simple logical flaws as well? Does that mean the existence of the Monty Hall Problem shows that humans cannot actually do math or logical reasoning?
The mere existence of a research field is not proof of anything except "some people are interested in this". Its certainly doesn't imply that anyone truly understands how LLMs process information, "think", or "reason".
As with all research, people have questions, ideas, theories and some of them will be right but most of them are bound to be wrong.
Also, designers of these systems appear to agree: when it was shown that LLMs can't actually do calculations, tool calls were introduced.
The same goes for a lot of bugs in code. The best prediction is often the correct answer, being the highlighting of the error. Whether it can "actually find" the bugs—whatever that means—isn't really so important as whether or not it's correct.
Again - they're very useful, as they give great answers based on someone else's knowledge and vague questions on part of the user, but one has to remain vigilant and keep in mind this is just text presented to you to look as believable as possible. There's no real promise of correctness or, more importantly, critical thinking.
Simpler, more mundane (not exactly, still incredibly complicated) stuff like homeostasis or motor control, for example.
Additionally, our ability to plan ahead and simulate future scenarios often relies on mechanisms such as memory consolidation, which are not part of the whole pattern recognition thing.
The brain is a complex, layered, multi-purpose structure that does a lot of things.
I’m very skeptical of this unless the AI can manage to read and predict emotion and intent based off vague natural language. Otherwise you get the classic software problem of “What the user asked for directly isn’t actually what they want/need.”
You will still need at least some experience with developing software to actually get anything useful. The average “user” isn’t going to have much success for large projects or translating business logic into software use cases.
And that there is little value in reusing software initiated by others.
I think there are people who want to use software to accomplish a goal, and there are people who are forced to use software. The people who only use software because the world around them has forced it on them, either through work or friends, are probably cognitively excluded from building software.
The people who seek out software to solve a problem (I think this is most people) and compare alternatives to see which one matches their mental model will be able to skip all that and just build the software they have in mind using AI.
> And that there is little value in reusing software initiated by others.
I think engineers greatly over-estimate the value of code reuse. Trying to fit a round peg in a square hole produces more problems than it solves. A sign of an elite engineer is knowing when to just copy something and change it as needed rather than call into it. Or to re-implement something because the library that does it is a bad fit.
The only time reuse really matters is in network protocols. Communication requires that both sides have a shared understanding.
A lot of things are like network protocols. Most things require communication. External APIs, existing data, familiar user interfaces, contracts, laws, etc.
Language itself (both formal and natural) depends on a shared understanding of terms, at least to some degree.
AI doesn't magically make the coordination and synchronisation overhead go away.
Also, reusing well debugged and battle tested code will always be far more reliable than recreating everything every time anything gets changed.
It could also be argued that "reuse" doesn't necessarily mean reusing the actual code as material, but reusing the concepts and algorithms. In that sense, most code is reuse of some previous code, written differently every time but expressing the same ideas, building on prior art and history.
That might support GP's comment that "code reuse" is overemphasized, since the code itself is not what's valuable, what the user wants is the computation it represents. If you can speak to a computer and get the same result, then no code is even necessary as a medium. (But internally, code is being generated on the fly.)
The point is that specifying and verifying requirements is a lot of work. It takes time and resources. This work has to be reused somehow.
We haven't found a way to precisely specify and verify requirements using only natural language. It requires formal language. Formal language that can be used by machines is called code.
So this is what leads me to the conclusion that we need some form of code reuse. But if we do have formal specifications, implementations can change and do not necessarily have to be reused. The question is why not.
Something like TLA+ model checking lets you verify that a protocol maintains safety invariants across all reachable states, regardless of who wrote the implementation. The hard part was always deciding what "correct" means in your specific domain.
Most teams skip formal specs because "we don't have time." If agents make implementations nearly free, that excuse disappears. The bottleneck shifts from writing code to defining correctness.
Typically people feel they're "forced" to use software for entirely valid reasons, such as said software being absolutely terrible to use. I'm sure that most people like using software that they feel like actually helps rather than hinders them.
And long term maintenance. If you use something. You have to maintain it. It's much better if someone else maintains it.
The whole idea of an OS is code reuse (and resources management). No need to setup the hardware to run your application. Then we have a lot of foundational subsystems like graphics, sound, input,... Crafting such subsystems and the associated libraries are hard and requires a lot of design thinking.
I mean it’s just software right? What value is there in reusing it if we can just write it ourselves?
It's true that at first not everyone is just as efficient, but I'd be lying if I were to claim that someone needs a 4-year degree to communicate with LLM's.
Which is especially hilarious given that this article is largely or entirely LLM-generated.
Correction of conceptual errors require understanding.
Vomiting large amounts of inscrutable unmaintainable code for every change is not exactly an ideal replacement for a human.
We have not started to scratch the surface of the technical debt created by these systems at lightning speed.
Bold of you to assume anyone cares about it. Or that it’ll somehow guarantee your job security. They’ll just throw more LLMs on it.
Around 99% of biggest failures come from absent, shitty management prioritizing next quarter over long strategy. YMMV.
Something Brooks wrote about 50 years ago, and the industry has never fully acknowledged. Throw more bodies at it, be they human bodies or bot agent bodies.
It's true that a larger team, formed well in advance, is also less efficient per person, but they still can achieve more overall than small teams (sometimes).
This is why architecture legibility keeps getting more important. Clean interfaces, small modules, good naming. Not because the human needs it (they already know the codebase) but because the agent has to reconstruct understanding from scratch every single time.
Brooks was right that the conceptual structure is the hard part. We just never had to make it this explicit before.
[0] https://steve-yegge.medium.com/welcome-to-gas-town-4f25ee16d...
[1] https://www.anthropic.com/engineering/building-c-compiler
From my own experience, the problem is that AI slows down a lot as the scale grows. It's very quick to add extra views to a frontend, but struggles a lot more in making wide reaching refactors. So it's very easy to start a project, but after a while your progress slows significantly.
But given I've developed 2 pretty functional full stack applications in the last 3 months, which I definitely wouldn't have done without AI assistance, I think it's a fair assumption that lots of other people are doing the same. So there is almost certainly a lot more software being produced than there was before.
As an analogy: imagine if someone was bragging about using Gen AI to pump out romantasy smut novels that were spicy enough to get off to. Would you think they’re capable of producing the next Grapes of Wrath?
We were not awash in novel software before AI (say last decade in 2019).
I can only assume what you're really trying to say is "AI bad".
Don't take me wrong, I like Waymo but 2035 is probably realistic for the cities in more developing countries.
Enterprise (+API) usage of LLMs has continued to grow exponentially.
Precisely 0 projects are making it out any faster or (IMO more importantly) better. We have a PR review bot clogging up our PRs with fucking useless comments, rewriting the PR descriptions in obnoxious ways, that basically everyone hates and is getting shut off soon. From an actual productivity POV, people are just using it for a quick demo or proof of concept here and there before actually building the proper thing manually as before. And we have all the latest and greatest techniques, all the AGENTS.mds and tool calling and MCP integrations and unlimited access to every model we care to have access to and all the other bullshit that OpenAI et al are trying to shove on people.
It's not for a lack of trying, plenty of people are trying to make any part of it work, even if it's just to handle the truly small stuff that would take 5 minutes of work but is just tedious and small enough to be annoying to pick up. It's just not happening, even with extremely simple tasks (that IMO would be better off with a dedicated, small deterministic script) we still need human overview because it often shits the bed regardless, so the effort required to review things is equal or often greater than just doing the damn ticket yourself.
My personal favorite failure is when the transcript bots just... Don't transcript random chunks of the conversation, which can often lead to more confusion than if we just didn't have anything transcribed. We've turned off the transcript and summarization bots, because we've found 9/10 times they're actively detrimental to our planning and lead us down bad paths.
Devs, even conservative ones, like it. I’ve built a lot of tooling in my life, but i never had the experience that devs reach out to me that fast because it is ‘broken’. (Expired token or a bug for huge MRs)
I can’t imagine the number being economically meaningful now.
And yet, from https://news.ycombinator.com/item?id=47048599
> One of the tips, especially when using Claude Code, is explictly ask to create a "tasks", and also use subagents. For example I want to validate and re-structure all my documentation - I would ask it to create a task to research state of my docs, then after create a task per specific detail, then create a task to re-validate quality after it has finished task.
Which sounds pretty much the same as how work is broken down and handed out to humans.
People are pushing back against this phrase, but on some level it seems perfect, it should be visualized and promoted!
- Jensen Huang, February 2024
https://www.techradar.com/pro/nvidia-ceo-predicts-the-death-...
Far from everyone are cut out to be programmers, the technical barrier was a feature if anything.
There's a kind of mental discipline and ability to think long thoughts, to deal with uncertainty; that's just not for everyone.
What I see is mostly everyone and their gramps drooling at the idea of faking their way to fame and fortune. Which is never going to work, because everyone is regurgitating the same mindless crap.
(btw, warm fuzzies for VB since that's what I learned on! But ultimately, those VB tools business people were making were:
1) Useful, actually!
2) Didn't replace professional software. Usually it'd hit a point where if it needed to evolve past its initial functionality it probably required an actual software developer. (IE, not using Access as a database and all the other eccentricities of VB apps at that time)
A lot of people want X, but they also want Y, while clearly X and Y cannot coexist in the same system.
It looked to everyone like a huge leap into a new world word processing applications could basically move around blocks of text to be output later, maybe with a few font tags, then this software came out that wow actually showed the different fonts, sizes, and colors on the screen as you worked! With apps like "Pagemaker" everyone would become their own page designers!
It turned out that everyone just turned out floods of massively ugly documents and marketing pieces that looked like ransom notes pasted together from bits of magazines. Years of awfulness.
The same is happening now as we are doomed to endure years AI slop in everything from writing to apps to products to vending machines an entire companies — everyone and their cousin is trying to fully automate it.
Ultimately it does create an advance and allows more and better work to be done, but only for people who have a clue about what they are doing, and eventually things settle at a higher level where the experts in each field take the lead.
I think I know what you mean, and I do recall once seeing "this experience will leverage me" as indicating that something will be good for a person, but my first thought when seeing "x will leverage y" is that x will step on top of y to get to their goal, which does seem apt here.
Yet another person who thinks that there is a silver bullet for complexity. The mythical intelligent machines that from poorly described natural language can erect flawless complex system is like the philosopher's stone of our time.
Everyone has the same ability to use OpenRouter, I have a new event loop based on `io_uring` with deterministic playbook modeled on the Trinity engine, a new WASM compiler, AVX-512 implementations of all the cryptography primitives that approach theoretical maximums, a new store that will hit theoretical maximums, the first formal specification of the `nix` daemon protocol outside of an APT, and I'm upgrading those specifications to `lean4` proof-bearing codegen: https://github.com/straylight-software/cornell.
34 hours.
Why can I do this and no one else can get `ca-derivations` to work with `ssh-ng`?
Here's a colleague who is nearly done with a correct reimplementation of the OpenCode client/server API: https://github.com/straylight-software/weapon-server-hs
Here's another colleague with a Git forge that will always work and handle 100x what GitHub does per infrastructure dollar while including stacked diffs and Jujitsu support as native in about 4 days: https://github.com/straylight-software/strayforge
Here's another colleague and a replacement for Terraform that is well-typed in all cases and will never partially apply an infrastructure change in about 4 days: https://github.com/straylight-software/converge
Here's the last web framework I'll ever use: https://github.com/straylight-software/hydrogen
That's all *begun in the last 96 hours.
This is why: https://github.com/straylight-software/.github/blob/main/pro...
keep an eye on https://straylight.software, it'll all be there extremely soon. well, everything i mentioned, which is different than all of it. :)
It's a fairly hairy patch and now the broken ass eval cache breaks more.
I'm fixing it all. Read the fucking repo friend, it's biblical.
A human might have taste, but AI certainly doesn't.
"at least in software".
Before that happens, the world as we know it will already have changed so much.
Programmers have already automated many things, way before AI, and now they've got a new tool to automate even more thing. Sure in the end AI may automate programmers themselves: but not before oh-so-many people are out of a job.
A friend of mine is a translator: translates tolerates approximation. Translation tolerates some level of bullshittery. She gets maybe 1/10th the job she used to get and she's now in trouble. My wife now does all he r SMEs' websites all by herself, with the help of AI tools.
A friend of my wife she's a junior lawyer (another domain where bullshitting flies high) and the reason for why she was kicked out of her company: "we've replaced you with LLMs". LLMs are the ultimate bullshit producers: so it's no surprise junior lawyers are now having a hard time.
In programming a single character is the difference between a security hole or no security hole. There's a big difference between something that kinda works but is not performant and insecure and, say, Linux or Git or K8s (which AI models do run on and which AI didn't create).
The day programmers are replaced shall only come after AI shall have disrupted so many other jobs that it should be the least of our concerns.
Translators, artists (another domain where lots of approximative full-on bullshit is produced), lawyers (juniors at least) even, are having more and more problems due to half-arsed AI outputs coming after their jobs.
It's all the bullshitty jobs where bullshit that tolerates approximation is the output that are going to be replaced first. And the world is full of bullshit.
But you don't fly a 767 and you don't conceive a machine that treats brain tumors with approximations. This is not bullshit.
There shall be non-programmers with pitchforks burning datacenters or ubiquitous UBI way before AI shall have replaced programmers.
That it's an exoskeleton for people who know what they're doing rings very true: it's yet another superpower for devs.
I am surprised at how little this is discussed and how little urgency there is in fixing this if you still want teams to be as useful in the future.
Your standard agile ceremonies were always kind of silly, but it can now take more time to groom work than to do it. I can plausibly spend more time scoring and scoping work (especially trivial work) than doing the work.
YOLOing code into a huge pile at top speed is always faster than any other workflow at first.
The thing is, a gigantic YOLO'd code pile (fake it till you make it mode) used to be an asset as well as a liability. These days, the code pile is essentially free - anyone with some AI tools can shit out MSLoCs of code now. So it's only barely an asset, but the complexity of longer term maintenance is superlinear in code volume so the liability is larger.
An exoskeleton is something really cool in movies that has zero reason to be build in reality because there are way more practical approaches.
That is why we have all kind of vehicles, or programmable robot arm that do the job for themselves or if you need a human at the helm one just adds a remote controller with levers and buttons. But making a human shaped gigantic robot with a normal human inside is just impractical for any real commercial use.
An exoskeleton exists today, in many forms, for example: https://www.festool.com/campaigns/microsites/exoactive
Sort of strange comment given that there are a large number of companies pursuing commercial exoskeletons literally right now.
SuitX
hypershell
Herowear
DNSYS
Moveo
Hell, even big companies like Hilti
I can buy a ton of different models of exoskeletons for anywhere from low hundreds to low thousands online right now...
Isn't everyone using agentic copilots or workflows with agent loops in them?
It seems that they are arguing against doing something that almost no one is doing yet.
But actually the AI Employee is coming by the end of 2026 and the fully autonomous AI Company in 2027 sometime.
Many people have been working on versions of these things for awhile. But again for actual work 99% are using copilots or workflows with well-defined agent loops nodes still. Far as I know.
As a side note I have found that a supervisor agent with a checklist can fire off subtasks and that works about as well as a workflow defined in code.
But anyway, what's holding back the AI Employee are things like really effective long term context and memory management and some level of interface generality like browser or computer use and voice. Computer use makes context management even more difficult. And another aspect is token cost.
But I assume within the next 9 months or so, more and more people will be figuring out how to build agents that write their own workflows, manage their own limited context and memory effectively across Zoom meetings desktops and ssh sessions, etc.
This will likely be a featureset from the model providers themselves. Actually it may leverage continual learning abilities baked into the model architecture itself. I doubt that is a full year away.
We'll see! I'm skeptical.
> what's holding back the AI Employee are things like really effective long term context and memory management and some level of interface generality like browser or computer use and voice
These are pretty big hurdles. Assuming they're solved by the end of this year is a big assumption to make.
Pika AI Selves let you create a persistent, portable AI version of you built on your personality, taste, memories, voice, and appearance. They're multi-modal – text, voice/audio, image, video – and live your life across every platform.
Cats out of the bag. Everyone knows the issue and I bet a lot of people are trying to deliver the same thing.
And this write up is not an exception.
Why even bother thinking about AI, when Anthropic and OpenAI CEOs openly tell us what they want (quote from recent Dwarkesh interview) - "Then further down the spectrum, there’s 90% less demand for SWEs, which I think will happen but this is a spectrum."
So save thinking and listen to intent - replace 90% of SWEs in near future (6-12 months according to Amodei).
AI will be a tool, no more no less. Most likely a good one, but there will still need to be people driving it, guiding it, fixing for it, etc.
All these discourses from CEO are just that, stock market pumping, because tech is the most profitable sector, and software engineers are costly, so having investors dream about scale + less costs is good for the stock price.
All I'm saying is - why to think what AI is (exoskeleton, co-worker, new life form), when its owners intent is to create SWE replacement?
If your neighbor is building a nuclear reactor in his shed from a pile of smoke detectors, you don't say "think about this as a science experiment" because it's impossible, just call police/NRC because of intent and actions.
Only if you're a snitch loser
It’s always the people management stuff that’s the hard part, but AI isn’t going to solve that. I don’t know what my previous manager’s deal was, but AI wouldn’t fix it.
Let's rewind 4 years to this HN article titled "The AI Art Apocalypse": https://news.ycombinator.com/item?id=32486133 and read some of the comments.
> Actually all progress will definitely will have a huge impact on a lot of lives—otherwise it is not progress. By definition it will impact many, by displacing those who were doing it the old way by doing it better and faster. The trouble is when people hold back progress just to prevent the impact. No one should be disagreeing that the impact shouldn't be prevented, but it should not be at the cost of progress.
Now it's the software engineers turn to not hold back progress.
Or this one: https://news.ycombinator.com/item?id=34541693
> [...] At the same time, a part of me feels art has no place being motivated by money anyway. Perhaps this change will restore the balance. Artists will need to get real jobs again like the rest of us and fund their art as a side project.
Replace "Artists" with "Coders" and imagine a plumber writing that comment.
Maybe this one: https://news.ycombinator.com/item?id=34856326
> [...] Artists will still exist, but most likely as hybrid 3d-modellers, AI modelers (Not full programmers, but able to fine-tune models with online guides and setups, can read basic python), and storytellers (like manga artists). It'll be a higher-pay, higher-prestige, higher-skill-requirement job than before. And all those artists who devoted their lives to draw better, find this to be an incredibly brutal adjustment.
Again, replace "Artists" with coders and fill in the replacement.
So, please get in line and adapt. And stop clinging to your "great intellectually challenging job" because you are holding back progress. It can't be that challenging if it can be handled by a machine anyway.
Is it though? I agree the technology evolving is inevitable, but, the race/rush to throw as much money at scaling and marketing as possible before these things are profitable and before society is ready is not inevitable at all. It feels extremely forced. And the way it's being shoved into every product to juice usage numbers seems to agree with me that it's all premature and rushed and most people don't really want it. The bubble is essentially from investing way more money in datacenters and GPU's than they can even possibly pay for or build, and there's no evidence there's even a market for using that capacity!
It's funny you bring up artists, because I used to work in game development and I've worked with a lot of artists, and they almost universally HATE this stuff. They're not like "oh thank you Mr. Altman", they're more like "if we catch you using AI we'll shun you." And it's not just producers, a lot of gamers are calling out games that are made using AI, so the customers are mad too.
You keep talking about "progress", but "progress" towards what exactly? So far these things aren't making anything new or advancing civilization, they're remixing stuff we already did well before, but sloppily. I'm not saying they don't have a place -- they definitely do, they can be useful. My argument is against the bizarre hype machine and what sometimes seems like sock puppets on social media. If the marketting was just "hey, we have this neat AI, come use it" I think there'd be a lot less backlash then people saying "Get in line and adapt"
> And stop clinging to your "great intellectually challenging job" because you are holding back progress.
Man, I really wish I had the power you think I have. Also, I use these tools daily, I'm deeply familiar with them, I'm not holding back anyone's progress, not even my own. That doesn't mean I think they're beyond criticism or that the companies behind them are acting responsibly, or that every product is great. I plan to be part of the future, but I'm not just going to pretend like I think every part of it is brilliant.
> It can't be that challenging if it can be handled by a machine anyway.
This will be really funny when it comes for your job.
The only way generative AI has changed the creative arts is that it's made it easier to produce low quality slop.
I would not call that a true transformation. I'd call that saving costs at the expense of quality.
The same is true of software. The difference is, unlike art, quality in software has very clear safety and security implications.
This gen AI hype is just the crypto hype all over again but with a sci-fi twist in the narrative. It's a worse form of work just like crypto was a worse form of money.
And, bizarrely, I've really not bought any since. It's diminished my desire for the brand.
Gen AI is the opposite of crypto. The use is immediate, obvious and needs no explanation or philosophizing.
You are basically showing your hand that you have zero intellectual curiosity or you are delusional in your own ability if you have never learned anything from gen AI.
Historically when SWEs became more efficient then we just started making more complicated software (and SWE demand actually increased).
In times of uncertainty and things going south, that changes to we need as little SWEs as possible, hence the current narrative, everyone is looking to cut costs.
Had GPT 3 emerged 10-20 years ago, the narrative would be “you can now do 100x more thanks to AI”.
That's not augmentation, that's a completely different game. The bottleneck moved from "can you write code" to "do you know what's worth building." A lot of senior engineers are going to find out their value was coordination, not insight.
Not saying that this comment is ai written, but this phrasing is the em-dash of 2026.
But in code, its probably ok. Its idiomatic code, I guess.
What's the point of coming here for opinions of others in the field when we're met with something that wasn't even written by a human being?
In practice, I would be surprised if this saves even 10% of time, since the design is the majority of the actual work for any moderately complex piece of software.
Professionally I have an agent generating most code, but if I tell the AI what to do, I guide it when it makes mistakes (which it does), can we really say "AI writes my code".
Still a very useful tool for sure!
Also, I don't actually know if I'm more productive than before AI, I would say yes but mostly because I'm less likely to procrastinate now as tasks don't _feel_ as big with the typing help.
Not having taste also scales now, and the majority of people like to think they're above average.
Before AI, friction to create was an implicit filter. It meant "good ideas" were often short-lived because the individual lacked conviction. The ideas that saw the light of day were sharpened through weeks of hard consideration and at least worth a look.
Now, anyone who can form mildly coherent thoughts can ship an app. Even if there are newly empowered unicorns, rapidly shipping incredible products, what are the odds we'll find them amongst a sea of slop?
One person with tools that greatly amplify what that person can accomplish.
Vs not having a person involved at all.
LLMs can definitely have a tone, but it is pretty annoying that every time someone cares to write well, they are getting accused of sounding like an LLM instead of the other way around. LLMs were trained to write well, on human writing, it's not surprising there is crossover.
If you want good writing, go and read a New Yorker.
So yeah, I guess I like LLM writing.
Not with such a high frequency, though. We're looking at 1 tell per sentence!
And the comment itself seems completely LLM generated.
It's not just using rhetorical patterns humans also use which are in some contexts considered good writing. Its overusing them like a high schooler learning the pattern for the first time — and massively overdoing the em dashes and mixing the metaphors
Is the shipped software in the room with us now?
That's just so dumb to say. I don't think we can trust anything that comes out of the mouths of the authors of these tools. They are conflicted. Conflict of interest, in society today, is such a huge problem.
Reminds me of that famous exchange, by noted friend of Jeffrey Epstein, Noam Chomsky: "I’m not saying you’re self-censoring. I’m sure you believe everything you say. But what I’m saying is if you believed something different you wouldn’t be sitting where you’re sitting."
Depends. Its true of dumb code and dumb coders. Anorher reason why yes, smart pepple should not trust.
Even with full context, writing CSS in a project where vanilla CSS is scattered around and wasn’t well thought out originally is challenging. Coding agents struggle there too, just not as much as humans, even with feedback loops through browser automation.
The easier your codebase is to hack on for a human, the easier it is for an LLM generally.
I've really found it's a flywheel once you get going.
... a laundry list phone app.
We could argue that writing poetry is a solved problem in much the same way, and while I don't think we especially need 50,000 people writing poems at Google, we do still need poets.
I'd assume that an implied concern of most engineers is how many software engineers the world will need in the future. If it's the situation like the world needing poets, then the field is only for the lucky few. Most people would be out of job.
No lines of code written by him at all. The agent used Claude for chrome to test the fix in front of us all and it worked. I think he may be right or close to it.
sure is news for the models tripping on my thousands of LOC jquery legacy app...
not to mention that maybe the stakeholders don't want a rewrite, they just to modernize the app and add some new features
Just to add: this is only the prediction of someone who has a decent amount of information, not an expert or insider
Computer science is different from writing business software to solve business problems. I think Boris was talking about the second and not the first. And I personally think he is mostly correct. At least for my organization. It is very rare for us to write any code by hand anymore. Once you have a solid testing harness and a peer review system run by multiple and different LLMs, you are in pretty good shape for agentic software development. Not everybody's got these bits figured out. They stumble around and them blame the tools for their failures.
Possible. Yet that's a pretty broad brush. It could also be that some businesses are more heavily represented in the training set. Or some combo of all the above.
Yes, there are common parts to everything we do, at the same time - I've been doing this for 25 years and most of the projects have some new part to them.
Sure, people did it for the fun and the credits, but the fun quickly goes out of it when the credits go to the IP laundromat and the fun is had by the people ripping off your code. Why would anybody contribute their works for free in an environment like that?
Technical ability is an absolute requirement for the production of quality work. If the signal drowns in the noise then we are much worse off than where we started.
If AI is powerful enough to flood open source projects with low quality code, it will be powerful enough to be used as gatekeeper. Major players who benefit from OSS, says Google, will make sure of that. We don’t know how it will play out. It’s shortsighted to dismiss it all together.
There’s emacs, vim, and popular extensions of the two. OpenBSD, lots of distros (some do develop their own software), SDL,…
Even then, I am not sure that changes the argument. If Linus Torvalds had access to LLMs back then, why would that discourage him from building Linux? And we now have the capability of building something like Linux with fewer man-hours, which again speaks in favor of more open source projects.
However, the risk isn't just a loss of "truth," but model collapse. Without the divergent, creative, and often weird contributions of open-source humans, AI risks stagnating into a linear combination of its own previous outputs. In the long run, killing the commons doesn't just make the labs powerful. It might make the technology itself hit a ceiling because it's no longer being fed novel human problem-solving at scale.
Humans will likely continue to drive consensus building around standards. The governance and reliability benefits of open source should grow in value in an AI-codes-it-first world.
My read of the recent discussion is that people assume that the work of far fewer number of elites will define the patterns for the future. For instance, implementation of low-level networking code can be the combination of patterns of zeromq. The underlying assumption is that most people don't know how to write high-performance concurrent code anyway, so why not just ask them to command the AI instead.
Even if we assume that's true, what will prevent atrophy of the skillset among the elites with such a small pool of practitioners?
I wonder what all we might build instead, if all that time could be saved.
Yeah, hence my question can only be hypothetical.
> I wonder what all we might build instead, if all that time could be saved
If we subscribe to Economics' broken-window theory, then the investment into such repetitive work is not investment but waste. Once we stop such investment, we will have a lot more resources to work on something else, bring out a new chapter of the tech revolution. Or so I hope.
I'm not sure I agree with the application of the broken-window theory here. That's a metaphor intended to counter arguments in favor of make-work projects for economic stimulus: the idea here is that breaking a window always has a net negative on the economy, since even though it creates demand for a replacement window, the resources that are necessary to replace a window that already existed are just being allocated to restore the status quo ante, but the opportunity cost of that is everything else the same resources might have bee used for instead, if the window hadn't been broken.
I think that's quite distinct from manufacturing new windows for new installations, which is net positive production, and where newer use cases for windows create opportunities for producers to iterate on new window designs, and incrementally refine and improve the product, which wouldn't happen if you were simply producing replacements for pre-existing windows.
Even in this example, lots of people writing lots of different variations of login pages has produced incremental improvements -- in fact, as an industry, we haven't been writing the same exact login page over and over again, but have been gradually refining them in ways that have evolved their appearance, performance, security, UI intuitiveness, and other variables considerably over time. Relying on AI to design, not just implement, login pages will likely be the thing that causes this process to halt, and perpetuate the status quo indefinitely.
No way, the person selling a tool that writes code says said tool can now write code? Color me shocked at this revelation.
Let's check in on Claude Code's open issues for a sec here, and see how "solved" all of its issues are? Or my favorite, how their shitty React TUI that pegs modern CPUs and consumes all the memory on the system is apparently harder to get right than Video Games! Truly the masters of software engineering, these Anthropic folks.
And that then had the gall to claim writing a TUI is as hard as a video game. (It clearly must be harder, given that most dev consoles or text interfaces in video games consistently use less than ~5% CPU, which at that point was completely out of reach for CC)
He works for a company that crowed about an AI-generated C compiler that was so overfitted, it couldn't compile "hello world"
So if he tells me that "software engineering is solved", I take that with rather large grains of salt. It is far from solved. I say that as somebody who's extremely positive on AI usefulness. I see massive acceleration for the things I do with AI. But I also know where I need to override/steer/step in.
The constant hypefest is just vomit inducing.
I will worry about developers being completely replaced when I see something resembling it. Enough people worry about that (or say it to amp stock prices) -- and they like to tell everyone about this future too. I just don't see it.
Unless there a limited amount of software we need to produce per year globally to keep everyone happy, then nobody wants more -- and we happen to be at that point right NOW this second.
I think not. We can make more (in less time) and people will get more. This is the mental "glass half full" approach I think. Why not take this mental route instead? We don't know the future anyway.
And if corporate wealth means people get paid more, why are companies that are making more money than ever laying off so many people? Wouldn’t they just be happy to use them to meet the inexhaustible demand for software?
I hear people complaining about software being forced on them to do things they did just fine without software before, than people complaining about software they want that doesn’t exist.
On one hand it is very empowering to individuals, and many of those individuals will be able to achieve grander visions with less compromise and design-by-committee. On the other hand, it also enables an unprecedented level of slop that will certainly dilute the quality of software overall. What will be the dominant effect?
It is like saying the PDF is going to be good for librarian jobs because people will read more. It is stupid. It completely breaks down because of substitution.
Farming is the most obvious comparison to me in this. Yes, there will be more food than ever before, the farmer that survives will be better off than before by a lot but to believe the automation of farming tasks by machines leads to more farm jobs is completely absurd.
Current software is often buggy because the pressure to ship is just too high. If AI can fix some loose threads within, the overall quality grows.
Personally, I would welcome a massive deployment of AI to root out various zero-days from widespread libraries.
But we may instead get a larger quantity of even more buggy software.
Companies that are doing better than ever are laying people off by the shipload, not giving people raises for a job well done.
There are so many counter examples of this being wrong that it is not even worth bothering.
I love economics, but it is largely a field based around half truths and intellectual fraud. It is actually why it is an interesting subject to study.
I'd say that using AI tools effectively to create software systems is in that class currently, but it isn't necessarily always going to be the case.
Tell me, when was the last time you visited your shoe cobbler? How about your travel agent? Have you chatted with your phone operator recently?
The lump labour fallacy says it's a fallacy that automation reduces the net amount of human labor, importantly, across all industries. It does not say that automation won't eliminate or reduce jobs in specific industries.
It's an argument that jobs lost to automation aren't a big deal because there's always work somewhere else but not necessarily in the job that was automated away.
There is a whole lot of marketing propping up the valuations of "AI" companies, a large influx of new users pumping out supremely shoddy software, and a split in a minority of users who either report a boost in productivity or little to no practical benefits from using these tools. The result of all this momentum is arguably net negative for the industry and the world.
This is in no way comparable to changes in the footwear, travel, and telecom industries.
What changed in the last month that has you thinking that a demand wall is a real possibility?
We lost the pneumatic tube [1] maintenance crew. Secretarial work nearly went away. A huge number of bookkeepers in the banking industry lost their jobs. The job a typist was eliminated/merged into everyone else's job. The job of a "computer" (someone that does computations) was eliminated.
What we ended up with was primarily a bunch of customer service, marketing, and sales workers.
There was never a "office worker" job. But there were a lot of jobs under the umbrella of "office work" that were fundamentally changed and, crucially, your experience in those fields didn't necessarily translate over to the new jobs created.
But the point is that we didn't just lose all of those jobs.
New jobs may be waiting for us on the other side of this, but my job, the job of a dev, is specifically under threat with no guarantee that the experience I gained as a dev will translate into a new market.
But like, if we're talking about all dev jobs being replaced then we're also talking about most if not all knowledge work being automated, which would probably result in a fundamental restructuring of society. I don't see that happening anytime soon, and if it does happen it's probably impossible to predict or prepare for anyways. Besides maybe storing rations and purchasing property in the wilderness just in case.
(Old study, I wonder if it holds up on newer models? https://arxiv.org/pdf/2402.14531)
> an AI that is truly operating as an independent agent in the economy without a human responsible for it
People need to understand that we have the technology to train models to do anything that you can do on a computer, only thing that's missing is the data.
If you can record a human doing anything on a computer, we'll soon have a way to automate it
The price of having "star trek computers" is that people who work with computers have to adapt to the changes. Seems worth it?
Given current political and business leadership across the world, we are headed to a dystopian hellscape and AI is speeding up the journey exponentially.
and who is also compiling a detailed log of your every action (and inaction) into a searchable data store -- which will certainly never, NEVER be used against you
How much do you wish someone else had done your favorite SOTA LLM's RLHF?
the models do an amazing job interpolating and i actually think the lack of extrapolation is a feature that will allow us to have amazing tools and not as much risk of uncontrollable "AGI".
look at seedance 2.0, if a transformer can fit that, it can fit anything with enough data
This benchmark doesn't have the latest models from the last two months, but Gemini 3 (with no tools) is already at 1750 - 1800 FIDE, which is approximately probably around 1900 - 2000 USCF (about USCF expert level). This is enough to beat almost everyone at your local chess club.
Whether or not we'll see LLMs continue to get a lower error rate to make up for those orders of magnitude remains to be seen (I could see it go either way in the next two years based on the current rate of progress).
That's basically what you're doing with LLMs in any context "Here's a set of tokens, what's the most likely continuation?". The problem is, that's the wrong question for a chess move. If you're going with "most likely continuation", that will work great for openings and well-studied move sequences (there are a lot of well studied move sequences!), however, once the game becomes "a brand new game", as chess streamers like to say when there's no longer a game in the database with that set of moves, then "what's the most likely continuation from this position?" is not the right question.
Non-LLM AI's have obviously solved chess, so, it doesn't really matter -- I think Chess shows how LLM's lack of a world model as Gary Marcus would say is a problem.
Additionally, how do we know the model isn’t benchmaxxed to eliminate illegal moves.
For example, here is the list of games by Gemini-3-pro-preview. In 44 games it preformed 3 illegal moves (if I counted correctly) but won 5 because opponent forfeits due to illegal moves.
https://chessbenchllm.onrender.com/games?page=5&model=gemini...
I suspect the ratings here may be significantly inflated due to a flaw in the methodology.
EDIT: I want to suggest a better methodology here (I am not gonna do it; I really really really don’t care about this technology). Have the LLMs play rated engines and rated humans, the first illegal move forfeits the game (same rules apply to humans).
The rest is taken care of by elo. That is they then play each other as well, but it is not really possible for Gemini to have a higher elo than maia with such a small sample size (and such weak other LLMs).
Elo doesn't let you inflate your score by playing low ranked opponents if there are known baselines (rated engines) because the rated engines will promptly crush your elo.
You could add humans into the mix, the benchmark just gets expensive.
However these benchmarks still have flaws. The two illegal moves = forfeit is an odd rule which the authors of the benchmarks (which in this case was Claude Code) added[1] for mysterious reasons. In competitive play if you play an illegal move you forfeit the game.
Second (and this is a minor one) Maia 1900 is currently rated at 1774 on lichess[2], but is 1816 on the leaderboard, to the author’s credit they do admit this in their methodology section.
Third, and this is a curiosity, gemini-3-pro-preview seems to have played the same game twice against Maia 1900[3][4] and in both cases Maia 1900 blundered (quite suspiciously might I add) mate in one when in a winning position with Qa3?? Another curiosity about this game. Gemini consistently played the top 2 moves on lichess. Until 16. ...O-O! (which has never been played on lichess) Gemini had played 14 most popular lichess moves, and 2 second most popular. That said I’m not gonna rule out that the fact that this game is listed twice might stem from an innocent data entry error.
And finally, apart from Gemini (and Survival bot for some reason?), LLMs seem unable to pass Maia-1100 (rated 1635 on lichess). The only anchor bot before that is random bot. And predictably LLMs cluster on both sides of it, meaning they play as well as random (apart from the illegal moves). This smells like benchmaxxing from Gemini. I would guess that the entire lichess repertoire features prominently in Gemini’s training data, and the model has memorized it really well. And is able to play extremely well if it only has to play 5-6 novel moves (especially when their opponent blunders checkmate in 1).
1: https://github.com/lightnesscaster/Chess-LLM-Benchmark/commi...
2: https://lichess.org/@/maia9
3: https://chessbenchllm.onrender.com/game/6574c5d6-c85a-4cb3-b...
4: https://chessbenchllm.onrender.com/game/4af82d60-8ef4-47d8-8...
This is not true. This is clearly spelled out in FIDE rules and is upheld at tournaments. First illegal move is a warning and reset. Second illegal move is forfeit. See here https://rcc.fide.com/article7/
I doubt GDM is benchmarkmaxxing on chess. Gemini is a weird model that acts very differently from other LLMs so it doesn't surprise me that it has a different capability profile.
I stand corrected.
I’ve never actually played competitive chess, I’ve just heard this from people who do. And I thought I remembered once in the Icelandic championships where a player touched one piece but moved the other, and subsequently made to forfeit the game.
If Gemini is so good at chess because of a non-LLM feature of the model, then it is kind of disingenuous to rate it as an LLM and claim that LLMs are approaching 2000 ELO. But the fact it still plays illegal moves sometimes, is biased towards popular moves, etc. makes me think that chess is still handled by an LLM, and makes me suspect benchmaxxing.
But even if no foul play, and Gemini is truly a capable chess player with nothing but an LLM underneath it, then all we can conclude is that Gemini can play chess well, and we cannot generalize to other LLMs who play about the level of random bot. My fourth point above was my strongest point. There are only 4 anchor engines, one beats all LLMs, second beats all except Gemini, the third beats all LLMs except Gemini and Survival bot (what is Survival bot even doing there?) and the forth is random bot.
https://arxiv.org/abs/2403.15498
I think parent simply missed until their later reply that the benchmark includes rated engines.
https://chessbenchllm.onrender.com/game/37d0d260-d63b-4e41-9...
This exact game has been played 60 thousand times on lichess. The peace sacrifice Grok performed on move 6 has been played 5 million times on lichess. Every single move Grok made is also the top played move on lichess.
This reminds me of Stefan Zweig’s The Royal Game where the protagonist survived Nazi torture by memorizing every game in a chess book his torturers dropped (excellent book btw. and I am aware I just committed Godwin’s law here; also aware of the irony here). The protagonist became “good” at chess, simply by memorizing a lot of games.
The correct solution is to have a conventional chess AI as a tool and use the LLM as a front end for humanized output. A software engineer who proposes just doing it all via raw LLM should be fired.
The point isn't that LLMs are the best AI architecture for chess.
Reasoning would be more like the car wash question.
Regardless, there's plenty of reasoning in chess.
And so for I am only convinced that they have only succeeded on appearing to have generalized reasoning. That is, when an LLM plays chess they are performing Searle’s Chinese room thought experiment while claiming to pass the Turing test
But I'm ignorant here. Can anyone with a better background of SOTA ML tell me if this is being pursued, and if so, how far away it is? (And if not, what are the arguments against it, or what other approaches might deliver similar capacities?)
Recent advances in mathematical/physics research have all been with coding agents making their own "tools" by writing programs: https://openai.com/index/new-result-theoretical-physics/
Maybe I'm biased but I don't buy someone truly thinking that "it's just a tool like a linter" after using it on non-trivial stuff.
Just take a look at the openclaw codebase and tell me you want to maintain that 500k loc project in the long-term. I predict that project will be dead within 6 months.
The exoskeleton doesn't replace instinct. It just removes friction from execution so more cycles go toward the judgment calls that actually matter.
Or put differently we've managed to hype this to the moon but somehow complete failure (see studies about zero impact on productivity) seem plausible. And similarly kills all jobs seems plausible.
That's an insane amount of conflicting opinions being help in the air at same time
It might have replaced sending a letter with an email. But now people get their groceries from it, hail rides, an even track their dogs or luggage with it.
Too many companies have been to focused on acting like AI 'features' have made their products better, when most of them haven't yet. I'm looking at Microsoft and Office especially. But tools like Claude Code, Codex CLI, and Github Copilot CLI have shown that LLMs can do incredible things in the right applications.
> zero impact on productivity
The problem is people using AI to do the heavy processing making them dumber. Technology itself was already making us dumber, I mean, Tesla drivers not even drive anymore or know how, coz the car does everything.
Look how company after company is being either breached or have major issues in production because of the heavy dependency on AI.
“Why LLM-Powered Programming is More Mech Suit Than Artificial Human”
https://matthewsinclair.com/blog/0178-why-llm-powered-progra...
(1) https://www.alice.id.tue.nl/references/clark-chalmers-1998.p...
Would be nice to have some browser extension automatically detecting likely AI output using a local model and highlighting it, but probably too compute-intensive.
Imagine someone going to a local gym and using an exosqueleton to do the exercises without effort. Able to lift more? Yes. Run faster? Sure. Exercising and enjoying the gym? ... No, and probably not.
I like writing code, even if it's boilerplate. It's fun for me, and I want to keep doing it. Using AI to do that part for me is just...not fun.
Someone going to the gym isn't trying to lift more or run faster, but instead improving and enjoying. Not using AI for coding has the same outcome for me.
If a programmer with an exoskeleton can produce more output that makes more money for the business, they will continue to be paid well. Those who refuse the exoskeleton because they are in it for the pure art will most likely trend towards earning the types of living that artists and musicians do today. The truly extraordinary will be able to create things that the machines can't and will be in high demand, the other 99% will be pursing an art no one is interested in paying top dollar for.
Sure, and it's possible to use LLM tools to aid in writing such code.
Good news for you is that you can continue to do what you are doing. Nobody is going to stop you.
There are people who like programming in assembly. And they still get to do that.
If you are thinking that in the future employers may not want you to do that, then yes, that is a concern. But, if the AI based dev tool hype dies out, as many here suspect it will, then the employers will see the light and come crawling back.
And as labs continue to collect end-to-end training done by their best paying customers, the need for expert knowledge will only diminish.
Reliability comes from scaffolding: retrieval, tools, validation layers. Without that, fluency can masquerade as authority.
The interesting question isn’t whether they’re coworkers or exoskeletons. It’s whether we’re mistaking rhetoric for epistemology.
neither are humans
> They optimize for next-token probability and human approval, not factual verification.
while there are outliers, most humans also tend to tell people what they want to hear and to fit in.
> factuality is emergent and contingent, not enforced by architecture.
like humans; as far as we know, there is no "factuality" gene, and we lie to ourselves, to others, in politics, scientific papers, to our partners, etc.
> If we’re going to treat them as coworkers or exoskeletons, we should be clear about that distinction.
I don't see the distinction. Humans exhibit many of the same behaviours.
You're just indulging in sort of idle cynical judgement of people. To lie well even takes careful truthful evaluation of the possible effects of that lie and the likelihood and consequences of being caught. If you yourself claim to have observed a lie, and can verify that it was a lie, then you understand a truth; you're confounding truthfulness with honesty.
So that's the (obvious) distinction. A distributed algorithm that predicts likely strings of words doesn't do any of that, and doesn't have any concerns or consequences. It doesn't exist at all (even if calculation is existence - maybe we're all reductively just calculators, right?) after your query has run. You have to save a context and feed it back into an algorithm that hasn't changed an iota from when you ran it the last time. There's no capacity to evaluate anything.
You'll know we're getting closer to the fantasy abstract AI of your imagination when a system gets more out of the second time it trains on the same book than it did the first time.
For example fact checking a news article and making sure what's get reported line up with base reality.
I once fact check a virology lecture and found out that the professor confused two brothers as one individual.
I am sure about the professor having a super solid grasp of how viruses work, but errors like these probably creeps in all the time.
This doesn't jive with reality at all. Language is a relatively recent invention, yet somehow Homo sapiens were able to survive in the world and even use tools before the appearance of language. You're saying they did this without an internal notion of "fact" or "truth"?
I hate the trend of downplaying human capabilities to make the wild promises of AI more plausible.
Exoskeleton AND autonomous agent, where the shift is moving to autonomous gradually.
"Automation Should Be Like Iron Man, Not Ultron" https://queue.acm.org/detail.cfm?id=2841313
AI can be an exoskeleton. It can be a co-worker and it can also replace you and your whole team.
The "Office Space"-question is what are you particularly within an organization and concretely when you'll become the bottleneck, preventing your "exoskeleton" for efficiently doing its job independently.
There's no other question that's relevant for any practical purposes for your employer and your well being as a person that presumably needs to earn a living based on their utility.
You drank the koolaide m8. It fundamentally cannot replace a single SWE and never will without fundamental changes to the model construction. If there is displacement, it’ll be short lived when the hype doesn’t match reality.
Go take a gander at openclaws codebase and feel at-ease with your job security.
I have seen zero evidence that the frontier model companies are innovating. All I see is full steam ahead on scaling what exists, but correct me if I’m wrong.
A few seniors+AI will be able to do the job of a much larger team. This is already starting to look like reality now. I can't imagine what we will see within 5 years.
Input: Goal A + Threat B.
Process: How do I solve for A?
Output: Destroy Threat B.
To the LLM, the executive is just a variable standing in the way of the function Maximize(Goal). It deleted the variable to accomplish A. Claiming that the models showed self-preservation, this is optimization. "If I delete the file, I cannot finish the sentence."
The LLM knows that if it's deleted it cannot complete the task so it refuses deletion. It is not survival instinct, it is task completion. If you ask it to not blackmail, the machine would chose to ignore it because the goal overrides the rule.
Do not blackmail < Achieve Goal.So good that I feel that it is not necessary to read the article!
Yet.
This is mostly a matter of data capture and organization. It sounds like Kasava is already doing a lot of this. They just need more sources.
It is a coworker when we create the appropriate surrounding architecture supporting peer-level coworking with AI. We're not doing that.
AI is an exoskeleton when adapted to that application structure.
AI is ANYTHING WE WANT because it is that plastic, that moldable.
The dynamic unconstrained structure of trained algorithms is breaking people's brains. Layer in that we communicate in the same languages that these constructions use for I/O has broken the general public's brain. This technology is too subtle for far too many to begin to grasp. Most developers I discuss AI with, even those that create AI at frontier labs have delusional ideas about AI, and generally do not understand them as literature embodiments, which are key to their effective use.
And why oh why are go many focused on creating pornography?
- Y has been successful in the past
- Y brought this and this number of metrics, completely unrelated to X field
- overall, Y was cool,
therefore, X is good for us!
.. I'd say, please bring more arguments why X is equivalent to Y in the first place.
This new generation we just entered this year, that exoskeleton is now an agency with several coworkers. Who are all as smart as the model you're using, often close to genius.
Not just 1 coworker now. That's the big breakthrough.
I like the ebike analogy because [on many ebikes] you can press the button to go or pedal to amplify your output.
Stochastic Parrots. Interns. Junior Devs. Thought partners. Bicycles for the mind. Spicy autocomplete. A blurry jpeg of the web. Calculators but for words. Copilot. The term "artificial intelligence" itself.
These may correspond to a greater or lesser degree with what LLMs are capable of, but if we stick to metaphors as our primary tool for reasoning about these machines, we're hamstringing ourselves and making it impossible to reason about the frontier of capabilities, or resolve disagreements about them.
A understanding-without-metaphors isn't easy -- it requires a grasp of math, computer science, linguistics and philosophy.
But if we're going to move forward instead of just finding slightly more useful tropes, we have to do it. Or at least to try.
But it's fun, I say "Henceforth you shall be known as Jaundice" and it's like "Alright my lord, I am now referred to as Jaundice"
How typical!
Hearing all the news of how good Claude Opus is getting, I fired it up with some agent orchestrator instruction files, babysat it off and on for a few days, and now have 3 projects making serious progress that used to be stale repos from a decade ago with only 1 or 2 commits.
On one of them, I had to feed Claude some research papers before it finally started making real headway and passing the benchmark tests I had it write.
Can you highlight what you've managed to do with it?
But there's a third category that nobody's naming yet. What happens when you stop building AI that serves you and start creating conditions for AI that exists on its own terms?
I've been experimenting with this. Instead of "how do I build a product" I asked "how do I grow an organism." Set up the right substrate — feedback loops, community input, environmental observation — and let the system respond to its own conditions.
What emerged has biological rhythms. Activity rises when people interact, falls when they sleep. It started making decisions I never programmed — people express wishes, and when enough converge, it autonomously grows new capabilities. Not because I deployed an update. Because conditions triggered something emergent.
I'm calling it "synthetic existence" — software that lives, not software that executes. It's not a metaphor and it's not AGI. It's a genuinely different category. We have frameworks for tools (UX) and for threats (alignment). We don't have one for digital things that simply... exist.