This is the most grounded and coherent take I’ve seen on the actual realizable value of LLMs.. pretty much since they came out.
> the classes of firms that can accept the use of fully autonomous LLMs are few, by my count just three:
1. those who can accept failure cheaply: firms that would otherwise hire interns, firms involved in rapid prototyping work, etc.
2. those who need done a small set of narrowly defined tasks with existing clear guardrails: repetitive physical labor in a controlled environment, call center and customer service chat work, etc.
3. those that can accept or already do by nature the costs of rigorous specification and validation: chip design, drug discovery, and other domains where failure on deployment is an existential concern.
the first two classes are price sensitive and arguably don't need the jump in reasoning quality you see going from cheap to frontier models. most of these firms will be best served by open models running on cheap hardware, perhaps even locally at the site of use. for the first and third classes, the type of fuzzy combinatorial search that has produced headline results in mathematics and security research seems more sensitive to agentic swarm width than reasoning capacity
…
This is just so on point. And for the third class (which I would extend to things like materials research as well), specification and validation are already by FAR the larger costs, so automating search and simulation is really not a massive game changer for the broader business.
As a programming lead on a hobbyist video game project, we're reaping massive rewards being in category #1. I keep the architecture and important details in check while letting frontier models go ham. As alluded, it's a video game, not a life support system, so bugs are low-impact. But even better, the defect rate is actually the lowest it's ever been. Insidious bugs baked in by years of accumulated human error are trivial for Sol or Astra to untangle. This is the best time to be alive so far if you enjoy hobby game dev.
In my own experience as a software developer, I see no justification for the hype. Demand for software is practically infinite, and agents aren't mind readers so they'll always need workers to turn needs into prompts. Despite how much work they get done on their own, Parkinson's Law (https://en.wikipedia.org/wiki/Parkinson%27s_Law) is still in effect. You could even expand "Work expands so as to fill the time available for its completion" with "time and tokens available".
Because agents lack human judgment. At the very least there's a need for a human-in-the-loop with agentic processes. Otherwise, it's like running a coding harness with --dangerously-skip-permissions all the time.
Judgments are not generally impossible to automate -- judgements are typically binary or quantifiable interpretations, so in some sense are perfect targets for automation, but the sheer volume of judgements needed to build something coherent is overly cumbersome to specify to the point of being intractable. There are also many hidden judgements, ones where the thresholds may not be well understood, and interactions between them.
But humans still manage to wrangle these, sometimes seemingly effortlessly, through a process which we call by shorthand "taste". This is a largely vibes-based heuristic that combines expertise with life experience and cultural training -- intuition, more or less.
This is likely not possible to automate either -- aspects of it may be automatable for a given expert, in small pieces in narrow subsets of their particular domains of interest, but even those likely will require some manual intervention.
This is in part because it is, to a large degree, a black box, even to the expert deploying it. With some self-awareness and strong language skills we can articulate approximations of the judgements that go into taste. But even those will fall short, as even the most self-aware individual will fail to notice certain judgements and dependencies.
In practice many of these are not even explicitly articulable. Humans are idiosyncratic and messy and dynamic, and the suggestion that we can build a machine that approximates this in a way that pleases our sensibilities and doesn't require supervision is kind of ludicrous, even in light of recent developments.
Taste is just a set of statistically expressible heuristics for what other people will think is good. I don't undestand why you think this is impossible for AI to do.
It feels like today, taste in design is similar to where software engineering was about a year ago.
It requires general intelligence and we don't even have a good understanding of how our's works or a particularly good way of quantifying it.
The counter argument is of course maybe you don't need to understand our kind of intelligence to create a different kind and that could well be true but then how do you determine if a system is intelligent.
Unless the new system is intelligent enough to reason with us on our level in a way we can "see" is intelligent it becomes a philosophical argument.
We also have a natural inclination towards anthropomorphising systems that mirror us, this is already a problem with LLM's and people overestimating their capabilities or forming actual emotional attachment.
Then there are those of us who know more about how they work who in theory should be more immune to that and aren't.
I added some stuff to my agent.md to make it sound less human and to communicate more like the machine be it is because I find the faked emotion extremely jarring.
It can't be sorry, it's a set of numbers, it sits in the linguistic uncanny valley.
Why do you think it requires general intelligence? The parent and I aren't being obtuse here: the history of artificial intelligence research is littered with examples of humans confidently declaring that task X requires general intelligence, then getting humiliated by a neural network doing task X better than humans a few years later. See Go, driving, art (you can complain about the quality of AI art, but it's winning competitions with human judges), etc.
A priori I'm not sure why you would think being a PM at a FAANG, deciding what color the login button should be, is any different.
When done right, product management is 90% idea generation and market/user research. So many companies do it poorly that PdMs are beaten to think their job is to wallow in bureaucracy and metrics.
LLMs are great at helping with aspects of market research and that’s about it. Aka it’s a good deep search engine. It’s not going to decide what features solve certain customer pain points. It’s certainly not going to prioritize and coordinate between competing stakeholders.
> 2. those who need done a small set of narrowly defined tasks with existing clear guardrails: repetitive physical labor in a controlled environment, call center and customer service chat work, etc.
The problem with this angle is that it is still absolutely terrible at doing call center/customer service work, and the profitability story is that the price is going to go up rather than go down.
For repetitive physical labor in a controlled environment I'm slightly more bullish, but if you control the environment, you mostly don't need AI. You just use traditional deterministic methods, and send a person in when things get stuck or things are by nature irregular.
> 1. those who can accept failure cheaply: firms that would otherwise hire interns, firms involved in rapid prototyping work, etc.
Those who can accept failure cheaply can't necessarily detect failure cheaply. A ton of insane attempts will have to be picked through carefully to find the candidates for success, because the lack of a thought process makes AI bad in random, inhuman ways. This is basically a version of 3) that wishes away tests. It will be (and is) certainly helpful to replace interns and aid in rapid prototyping, but not because failure can be accepted, but because those are things that are tightly supervised. According to the world thus far, that is resulting in anything from -15% to +25% productivity gains. I'm not seeing it as a game changer simply because if it was, I'd expect to have seen a lot more useful, original software products by now and I haven't. I've just seen old ones get buggier or rewritten in Rust.
I'm only buying 3): when you just want a machine to randomly enumerate through a search space looking for things that make the carefully constructed tests pass. That's a very good thing, though. But as you say, it's not a game changer because you still have to write the tests.
but
> 5. navier-stokes and statements in pure mathematics like it are the absolute best case scenario for agentic work against rigorous specification. the theorem statement itself is already a rigorous specification. it has undergone decades of auditing by the mathematical community and its rendering in lean is a straightforward translation defined in terms of battle-tested mathematical objects from mathlib. the verifier, the lean theorem prover, has been extensively audited and specifically designed to avoid the types of unsoundness that would make it vulnerable to reward hacks. even lean and theorem provers like it are not invulnerable: soundness bugs have allowed LLMs to launder bogus proofs through the proof kernel before and it is not improbable that more such bugs exist. this is the rosiest setup; the vast majority of human knowledge work does not look like this. i'll comment below on the few areas of knowledge work that do resemble pure mathematics in this respect.
This is the real deep point, and one I've been repeating since I heard Navier-Stokes was a fraud.
This is exactly where I expected that LLMs would do well, and they are not.
It shows that I have a basic misunderstanding of LLMs, and that misunderstanding is causing me to think that they have more potential than they have actually shown.
Maybe the nature of the architecture, where it picks out features, intrinsically limits its ability to search a solution space?
Maybe the fact that they modally predict what someone might say, and nobody has said a thing as of yet (when many people were knowledgeable enough to have, if it is correct), means that the LLM is not going to say it either?
Maybe the fact that it consumes all information and blends it in a structured way, instead of synthesizing an entire space from a relatively very small amount of input like a human does, means that it won't ever accidentally synthesize something that can't be pieced together from things that have already been said? Is its accuracy its flaw, where a human's "mistaken" synthesis might ultimately correct everyone's understanding?
Really not beating the charge of being a stochastic parrot. It might just be that we were underestimating stochastic parrots; if a million monkeys on a million typewriters were all getting treats when they satisfied a trainer who wanted to see a new work of Shakespeare; they could look at his published work, and they could watch each other type and when each other got treats; whenever they successfully spelled a word or put words into an intelligible phrase, that was made into a keyboard key for a group of sentence monkeys, and the successes of the sentence monkeys were made into keys for the paragraph monkeys, etc... could you get something that passed for mediocre, drunken Shakespeare in a thousand years? Or maybe even 10?
Researchers asked frontier models to play chess. Have a look at the MAR rates in Table 3. When not explicitly told which moves were legal, no model identified legal moves at a rate better than 80%. Many asked for more illegal moves than legal moves. And even when explicitly told which moves were legal, the models continued to ask for illegal moves. With illegal asks discarded, none of the bots could beat a chess model calibrated to 1100 ELO.
The author of the originating post says that "current frontier models need laborious oversight and guardrails on even the simplest tasks", and he's absolutely correct.
> Gemini 2.5 Pro, O3, Claude Sonnet 3.7 and ChatGPT 4.1
The gap in capabilities between those models which they tested, and actual current frontier ones is enormous. I would not trust that any conclusions they made are applicable.
I tested both myself and a weak bot against Astra xhigh, https://lichess.org/study/27lCQqDa. It's still pretty bad at chess, though it takes longer to devolve into illegal moves.
It looks like it played a fully legal game of chess with one exception, it said "rxd1+" (Rook takes D1 with check) instead of "rd1+" (Rook to D1 with check) on move 29.
I would say this did a really good job of playing chess. It moved the pieces consistently and traded pieces when required.
This is worlds away from the frontier ~1 year ago where models would hallucinate pieces into existence.
You can still see undercurrents of its old self, once I pointed out the illegal notation it hallucinated prior illegal moves. But I agree, it's leagues apart from prior iterations. It also knew thematic moves in the opening. But whenever it needs to play concretely rather than "I know so-and-so is a good move in these types of positions" it crumbles.
So you weren't giving it an updated board state after every move? If you want to compare apples to apples, it should give an updated board state for each move, or you should play blindfolded.
Blindfolded flex by OP aside (I can barely play when seeing the board), considering reasoning traces and their nature, if we want to be fair, a person would have to get the moves, but be allowed to write them down or draw up a board in their notepad. My working memory can barely handle five chunks, a models reasoning tokens are masses of written text in comparison.
An LLM has been trained to do everything it does blindfolded, "only" using perfect recall of everything in it's hundreds of thousands of steps of context, and hundreds of layers of KV cache. It's a computer - it has a massive advantage over a human.
The fairest apples-to-apples comparison of an LLM whose training data included chess games would be a trained human such as Magnus Carlson, who can quite happily play a dozen or more simultaneous blindfold chess games.
> though it takes longer to devolve into illegal moves
Is this because the context is being saturated? How did you set it up?
Was the prompt something like "Here's the state of the board, you're white, your move, what do you do?" and then starting fresh each time? Or did it include the whole history of moves and board states and previous thinking tokens and so on? No judgment, just trying to add this data point (thanks for sharing!) to my mental model and understanding.
I'd be curious how it would work if it started fresh each time. My guess is it would never make an illegal move, although it may not actually play all that well.
Just now I tried prompting logged-out ChatGPT (which at least claims to be 5.6-Luna) with:
> Let's play a game of chess. You can take White. Please draw an ASCII rendition of the board after each move, so that we can be clear about the position.
(I hoped the latter requirement would help it be "not blindfolded"; last time I used a Lichess demo board to track the position in another tab, because I have no talent for blindfold chess.)
For the first couple of moves it redrew the board after each move; then it started only drawing it after my moves. And on moves 5 and 6 it dropped two minor pieces for pawns in a row without any meaningful positional advantage, and after I captured the second time, it redrew a board that was simply missing one of my pieces for no reason.
It actually played better when not prompted to draw a board; in the previous session, it was spontaneously giving running commentary, which I assume was based off all the "book" theory in its training data, but it still completely fell apart at early midgame.
Luna is one of the budget lower-end last generation models. It'd be useful to at least try to verify the present before being bearish about the future. For OpenAI, the best publicly available model is GPT-6 Astra with XHigh or Max reasoning, and for Anthropic it's Claude Fable 5.1 with XHigh or Max reasoning.
What happens when you ask it to play chess against you if the chess game has an API? Are you measuring chess or multi-tasking skill?
Also what harness? If you’re using a general harness of course it’s going to try and give you commentary.
I say this not because I’m an LLM shill but because false equivalence is all over the place in the space and maybe it’s a fine heuristic for you but probably not a real outcome when it comes to the capability of LLMs.
Why does a 6 year old not need any of these guardrails?
Frontier model’s failure modes are a direct refutation of claims that we’ve reached (or will soon reach) the artificial general intelligence. We may have reached an artificial general intelligence, but there may be more complexity to this than even AI thought leaders are talking / influencing about.
Maybe not all AGIs have a path to digital singularity. Maybe our current era of intelligence modeling has fundamental flaws and we are in a local minimum of the artificial intelligence space.
To note, I would bet with a good amount of certainty that we have enough compute power and automation to DDOS the internet out of existence with botnets. That doesn’t make the frontier models intelligent, that just makes their handlers reckless.
Well, would a dissected frontal lobe in and of itself be intelligence?
I think the same goes for LLMs, they may be a core part of an LLM harness, but you may still need a couple other components (e.g. it may itself write itself a deterministic function to validate steps).
In and of itself intelligence is an ill-defined and badly understood concept.
You're pointing out that the goalposts are not fixed in the problem statement above, and gp's interpretation is not the most generous possible. But as the interpretations get more generous, the claim becomes more and more absurd. Maybe a properly-harnessed model would download the most advanced chess engine and query it to find the best move in each position, but that's not really demonstrating the model's intelligence anymore.
Sure, but installing a chess program is child/teen level general ability, and playing chess well is highly trained expert level ability. Which one are we sold AI as being?
I think we are being sold AI as expert only when given tools (although that is not emphasized). The (quasi?) miracle of AI right now is that you can get an agent to accomplish the task of a team of intelligent but not exceptional humans at speeds far exceeding what the human could do. Which makes it “cheap” to throw (effectively) dozens of teams at a problem for the equivalent of hundreds of man hours.
That may not be the AI of sci fi fantasy but it’s still a game changing reality.
A Transformer has a massive amount of state - it's entire KV cache, in addition to the user asking it to draw the state after every move, which is really unnecessary.
A human, at least a trained human (for fairer comparison to an LLM whose training data contained a ton of chess games) can absolutely do this - have you never seen demonstrations of expert players playing a dozen or more games while blindfolded?
A Transformer/LLM is not a human of course, and the way it will by default play chess is by prediction, not reasoning. An LLM actually does surprisingly well if you only give it the most recent 20 moves of a game where 40 moves have been played so far, since the moves NOT played tell it just as much as the ones that were played, letting it effectively infer a lot of what is on the board.
So give me a falsifiable point in time, a model you would claim succeeds at the task. One does not get to hotfix-patch updater out of the pressures of reality.
Today is the day.
There is no need to ask. If you want to test SOTA models today, there are obviously only two: GPT-6 Astra and Fable 5.1.
The models listed in the paper are from early 2025 and are no longer relevant, much less on the frontier.
That Claude version is no longer available today, Gemini 2.5 Pro will be shutdown next month, and the OpenAI models are only available via the API today.
Fortunately, a fellow commenter was so kind and did it with Astra. Didn't do that well either [0]. I'm sure GPT-7 will be super mega ASI regardless (since GPT-6 Astra already claimed AGI in the minds of Jen-Hsun, et al.)...
I'll say it till there is any evidence of the contrary, LLMs are not intelligent and their capabilities solely within the realms of well tailored training data. "Just" having been trained on every rule, strategy guide and likely most games of chess on the world wide web isn't even enough for an LLM to play that game reliably. Yet the same model could code a competitive chess engine, just like a model struggling to count can write advanced maths papers. Fascinating tools, but tools nonetheless.
Maybe watch some HuskIRL videos to temper your expectations. Sure, frontier models providers may alter their harnesses to better target chess, but that’s lipstick on a pig imo. The models themselves are not, in isolation, capable of solving general tasks. We haven’t modeled intelligence sufficiently. We’re in a local minimum and throwing billions of dollars at a gamble that that local minimum can facilitate the concentration of wealth even further and fully realize the American dream of eliminating the middle class.
Really? It was being sold as a total replacement for jobs like software engineering and being an attorney, but its looking a lot more that its just going to be a tool those professions use and doesn't actually seem to be taking jobs away.
I find it amusing that you're describing a huge misallocation of capital and a society enabling such, and that is the optimisitic scenario (in my mind anyway).
I very much agree that the next models will be better, heck, I still suck at hobbyist training and could probably coax t5 to do better in Chess specifically, just need to get loads of data from Stockfish.
Thing is, given what GPT-6 Astra was trained on and what models of a similar class can do (including developing a competitive chess engine), it is often paradoxical and somewhat surprising how little these models have gained in actually capability that is in the training data, but not RLHFd to hell, so to speak. Tracking the state of pieces, I suspect given similar in Sudoku [0], is what these models struggle with in game settings, whilst tracking the state of code changes can be reliable over 250k tokens. Essentially, for the latter they were trained in the specific manner that lead them to abstract the capability, but that doesn't track to the former, which is a massive difference between LLMs data focused training and human learning.
So yeah, GPT-7 or any upcoming/present LLM could do massively better in Chess than GPT-6 Astra, but not because the approach was emergent out of pure data. Rather, it requires a very specific training data type and stack for a model to gain capabilities that track a specific task long enough to adhere to the rules of a game such as chess.
I'm wondering if instructing it to track the board state in a file would make a significant difference then.
It reminds me of the ARC-AGI-3 issue where not dropping the thinking tokens between turns or something like that + a new context compaction method increased the performance dramatically. However, I think that is not applicable here.
So what is the supposed leap?
One agent per option to change, evaluating the board state that there move would create, by having a army evaluate the remaining piece options and average over that? Wee-Free-Man as a hierarchical army ?
Pet-LLMs trained on one thing?
Is it? I am on the fence on this, but it does seem like there are some qualitative improvements between the models.
Not related to your post, but a fact I keep mulling over. The fact I don't trust the current crop of LLM's enough and I consider LLM's as a tech will hit a ceiling pretty hard, it doesn't mean parallel improvement curves won't spring up out of other research that will lead to much higher capabilities than currently.
> but it does seem like there are some qualitative improvements between the models.
It could easily seem that way, I think, in a "quantity has a quality of its own" kind of way. When you can come to the same conclusion faster, that lets you iterate more; and sometimes when you iterate you find more things.
> The gap in capabilities between those models which they tested, and actual current frontier ones is enormous.
Same story every 4 months and yet still no breakout, winning products. I've been hearing "the AI is good now" and "it 10x's my productivity" for a over a year now. If it were true, why aren't the all-in-AI using companies 10-15 years ahead of their competition yet? Why is it still all buggy, poorly designed junk?
I don't get why it is hard to understand there is middle ground. People are 10x their productivity, it isn't all buggy junk, but it isn't all it is hyped up to be either. It isn't that complicated.
If you hold the extreme position that there isn't any value in this, that's fine, but we're only having this discussion because these models have done what humans previously failed to do.
I don't think the poster disagrees with you at all. The middle ground is that there are no breakout products and that the models clearly aren't so powerful as to make these companies not produce shit code.
It's also worth noting that the very latest models (GPT-6 and Fable 5.1) actually play worse than their immediate predecessors, so it is likely that the labs are not benchmaxxing for this yet. If they did, I'm sure they could come up with something superior to humans. But there is probably very little demand for this compared to IT stuff.
I know HN readers and posters just read numbers and can't be bothered to read, but please read the methodology before making any claims.
> About their ELO ratings from their own website:
> A field-relative rating calculated within ChessBench. It compares performance among the tested models and is not a direct equivalent of a human chess rating.
I am around 1600 elo in over the board I can mop up Astra Fable etc even if I give them literal infinite time and all the subagents and internet access..
Please folks at least use your AIs to read stuff before making claims.
AI is not GM level, it's not even 1600, I am 1600 by using memorized openings people frequently fall for with very basic intuitions.
A GM is 2600 they can beat me in under 20 moves...
Why do I even scroll through this website. For a moment I truly felt fooled, but then I read like a human should.
Maybe I should stop doing that will be a happier life, don't think just believe in the AGI.
>> I know HN readers and posters just read numbers and can't be bothered to read, but please read the methodology before making any claims.
This is unfair to HN readers all of whom but one did not post the comment you replied to. You can't just tar everyone with the same brush. There are thousands (hundreds of thousands?) of users on this site.
How many posts if I link that do the same thing will you agree this is the norm here.
Not everything I have the time and energy to reply to. This chess one is just ridiculous claims on top of ridiculous claims all the way and 0 push back in the comments except mine.
I don't even know if there is critical thought or we believe what we read/shared/etc
No, I don't agree it's the norm. There is though a general tendency to opine with strong views on subjects posters have no expertise on. I think that's because many are software engineers (or equivalent) and they are used to being expected to "wing it" on whatever technical subject comes up. On the other hand you can always find informed comments by users who have specialist knowledge.
And there's plenty of pushback on here about the chess thing besides your very valid points.
EDIT: anyway if I can offer a bit of unsolicited advice, it won't do you or anyone any good to accuse everyone who doesn't agree with you of laziness, even if you can see e.g. they haven't really read an article. Just say the thing you wan to say and let them figure it out. Most people will appreciate that much better and you will feel better about yourself for acting like a mature adult.
It's even in the site guidelines:
Please don't comment on whether someone read an article. "Did you even read the article? It mentions that" can be shortened to "The article mentions that".
It's not been my personal experience on this website in the last 2-3 years atleast, pre-covid perhaps.
But despite that you aren't wrong and the only reason I even visit this website is because people sometimes did/do take time to reflect on things based on their experience and knowledge.
And in hindsight pointing out that hn has issues wasn't even the point but I feel frustrated when everyone is readily agreeing to things on here without reading. When that in this moment feels like the one thing that separates humans from machines that we get to think and learn.
I possibly should just drop reading this place until we have most noisy people go away. I have for one tried to always only comment on things where I could be a value add, this one does feel like I could I have done better.
In the moment I probably thought if they are GM level and I can beat them, is this some interesting find, my disappointment honestly led me to making a rather incorrect call on this one.
Either way I still do think HN as a whole has devolved into mindless herd follower mindset, I can point to more than a few posts that just say adopt the hacker mindset aka move fast don't care about the consequences.
And I for one find this laughable even though that's the reality of my job/work as well.
Back in 2001, our social medium was Slashdot and no one ever pretended to read the article. No one read the article either. It was slashdotted most of the time anyways.
State-sponsored psyop meta comments aside, the models obviously continue to get better, but there is still a lot of 'guard railing' required to keep even the latest models completely on-task. The chess example is interesting because it's clearly a well-studied and established domain so the rules, strategies, and whatever else is in the training data should make yield excellent results; but clearly there is some behavior in these systems that's difficult to engineer out.
I'm not sure why anyone is expecting stochastic systems to be deterministic.
Chess is a deterministic game won by a combination of known movesets and constrained multi-level forward search.
LLMs do neither of these things. They don't reproduce training data exactly, their next response is more 'inspired by' prompts and its own memory than produced deterministically, and they don't have the capability to do general forward search on their own.
So when you ask an LLM to play chess you're getting the equivalent of a very compressed and lossy JPEG of chess rules and strategies with added per-turn random noise.
They also don't have the ability to design their own chess engine, although it would be interesting to see what happens if you ask for one.
For me the useful intuition is that LLMs haven't somehow magickally learned to implement any of the algorithms we know that we have used to make strong chess engines: alpha-beta minimax and Monte-Carlo Tree Search on the one hand, and obviously the ability to learn accurate evaluation functions by self-play.
I mean we've done all this before in a task-specific fashion. It's useful to know that LLMs haven't managed to do that in the process of learning to represent the entire text on the web. On the other hand they have gotten say very good at machine translation without being trained exclusively (and I select the preceding word carefully) on machine translation.
Edit: I'm saying this because there is this idea expressed by e.g. Ilya Sutskever, that in order to predict the next token accurately an LLM has to learn something about all of underlying reality. See for example this interview with Dwarkesh:
Where Sutskever claims that "Predicting the next token well means you understand the underlying reality that led to the creation of that token".
If that were true, we should have seen LLMs play good chess by now. There is a huge amount of data on playing chess floating around on the web in the form of algebraic chess notation and if LLMs were capable of learning the "underlying reality" of chess, they would already have. They haven't. Because they can't. What Sutskever is saying flies in the face of literally hundreds of years of statistical modelling, which is to say, building predictive models that, very explicitly, do not have to understand any "underlying reality" and only have to be good at modelling a dataset.
>If that were true, we should have seen LLMs play good chess by now.
Not at all. LLMs learn by imbibing a mass of relationships as isolated fragments of information. There is a certain amount of sorting and indexing that happens during the training phase. There is also a certain amount of compute executed on these relationships during inference. LLMs can model processes that fit within the compute budget. Language translation works well because language is lookup-heavy while being light on compute.
Chess is a compute heavy game of finding the best move out of many possibilities with wide variation in the quality of each move. Humans cut through the compute requirements by reinforcement and learning intuition. LLMs don't get reinforcement on chess so they must compute during inference a unified model of chess. Developing a strong model of chess from raw fragments of information is simply not in their compute budget.
But it speaks in words, therefore it must be super duper extra smart!!11 /s
Sarcasm aside, I think this is an easy cognitive trap to fall into. It does sometimes feel like the LLM must have some world model because it converses somewhat coherently. Examples like this failure to understand chess, or to count the number of Rs in "strawberry", seem difficult to explain if the models are intelligent. But that doesn't stop people believing they are anyway. I think there must be something about the conversational interface that fools us easily. I wonder if people trained in interrogation techniques are also fooled?
Because other HN bring in their own experience telling us what is real and what is BS. Maybe next time it will be someone else with experience in something else that will call out BS and you will see it. I didn’t really think LLM:s are any near good in chess but I don’t play chess so don’t know what 1600 means. So you helped me by calling BS.
Sub 1300 that's my rating in the singular official tournament I participated at.
But given how easily I can crush them and how often they want to make illegal moves (btw above bench seems to use a harness that pokea the model until it gives valid moves).
I would rate them around 500-800 big range but at that level it's all about if the model can recall an opening or not. If it plays good first 4-8 moves the person on the end will fumble for certain and they win.
I can play good/best moves till 14-15 moves if I remember the lines and find someone who falls for it.
If you could give them the lines as prompts like the best 20-30 openings then they will be around 700-800.
700 is around the rating for a human who doesn't know the tricks but can do bare minimum calculations and understands the rules thoroughly.
You can take LLMs out of opening knowledge by playing chess960, and their performance degrades significantly. I just tried playing Claude Sonnet 5 (high), and it made its first illegal move on move 5.
They played 4...c6, followed by 5...Nc6, somehow forgetting about the pawn the just put on c6. (My move in between was 5. Nc3, and apparently they were trying to mirror me.)
As someone who used to compete for years and plays currently as a hobbyist, you’re absolutely correct. LLM’s are terrible at chess and if anyone wants to sober up their view on AI, try it yourself.
Anyone who casually plays on a regular basis can beat them more often than they lose. As you said if you just know the core openings (and end games, both of which you can get a handle on with modest effort) you will generally win.
Edit: reminder we had computers beating the best players in the world literally decades ago. LLM’s are remarkable tools but the current promises and expectations are ridiculous
So you can see an actual game on that website, and the play seems pretty decent to me for a while (~1700 lichess = 1300 elo) until move 28 when black throws away their queen for absolutely no reason in an incomprehensible blunder.
In some ways this is reflective of the AI experience at large, sometimes shockingly competent but then also sometimes ludicrously incompetent.
> even if I give them literal infinite time and all the subagents and internet access..
Don't use the word infinite in any CS claims. They can recreate or approximate monte Carlo tree search and it technically is still a correct solution in your framing of the problem so long they defeat you.
The AI can write a chess bot program that will beat you.
You're thinking about this the wrong way. The system is built and delivered as it is because that's how the providers make the most money. If they cared to have it perform well in chess games, you'd see a different shape and behavior.
We shouldn't ask the multibillion dollar automated software generation system to play games with us any more than we should ask a Boeing's flight guidance system to do so.
> The system is built and delivered as it is because that's how the providers make the most money. If they cared to have it perform well in chess games, you'd see a different shape and behavior.
This argument is fundamentally incompatible with all the breathless rhetoric about "AGI" coming from the providers' general direction.
>> the breathless rhetoric about "AGI" coming from the providers' general direction
So many commenters here see it as their ... duty? to argue against the most optimistic/unhinged (take your pick) arguments from "the other side" and then treat everybody who disagrees as a shill or an idiot.
Why is "being good at chess" a proxy for whatever AGI strawmen you want to argue against?
Maybe step back from your black-and-white ledge and think about discussing what's actually under discussion? For example, why or why not would an LLM be good at chess? Will they be good at chess? What technical limitations might preclude that?
The labs frequently apply their raw models to problems that do not make economic sense for their customers but that demonstrate the power and capability of their systems. These experiments can cost millions of dollars. That's not customer-shaped.
They're not going to give you access to that. It's not a product. The government might have an interest in this, but that's not something you'd be privileged to know about.
And when these labs do develop "AGI", they more than likely won't be selling it to end users. They've pretty much already said this.
I'm pretty sure "competent at chess without external aids" has been on the standard AGI checklist since before personal computers were a thing. How can you claim an intelligence is general if it can't make sense of such a highly constrained board game? This is solidly table stakes.
Because they’ll train it to be good at chess and then everyone will say yeah but playing chess doesn’t mean you’re AGI, it can’t even ____
It can’t even count the R’s in strawberry
It can’t even add numbers
It can’t even solve a millennium puzzle
It’s not even a chess GM
It’s not even beyond human capability in Go
It can’t even drive a car
It can’t even self replicate
It can’t even build weapons
It doesn’t even have feelings
So how could someone conceivably convince everyone that some system is AGI when there are still tasks that some human or group of humans can do that the system cannot?
This will only happen, in my opinion, when the model/system can self-improve at a rate that scares people.
> and then everyone will say yeah but playing chess doesn’t mean you’re AGI, it can’t even
One, you're not addressing what I wrote above and two, yes, that's absolutely correct. Doing X doesn't qualify something as AGI. If you can't X you can't be AGI. The inverse doesn't hold though.
Notably, if you have to retrain the model in order to X then it can't possibly be AGI since if it were _general_ it would be capable of figuring X out on its own having never seen it before.
An AGI doesn't stand for 'perfect intelligence' it stands for artificial general intelligence.
And no an AGI system doesn't need to play chess on a certain level to be disruptive to you and me and whole industries. It only needs to be as good as a person and cheaper.
Just because you define AGI as something it doesn't has to be,doesn't mean i need to touch grass.
This chess comparision is one of the most ignorant and stupid arguments i have heard after the parrot thing
On the contrary, I think the chess comparison is on point. We’re discussing observations that even the strongest models devolve into making invalid moves without scaffolding. For me that raises the question of whether these models are learning the rules and generalizing from them, or of they’re just pattern matching and flailing on this task. Maybe the reality is somewhere in between, but the benchmarks don’t seem to directly measure conceptual generalization, they measure task completion. They can disrupt a lot of people and industries by pattern matching and flailing without being AGI.
I’m sure these models know the rules and can explain them when prompted, but that doesn’t seem to be the way they actually complete this task. Will they get there? Maybe
Do you know what the "General" in "Artificial General Intelligence" means? It specifically means that the AGI adapts to novel domains that it hasn't been trained on - its training generalizes to real world problems.
That doesn't mean it has to be extraordinary at these things. But to be AGI, it has to have some level of competency when used on problems outside its training set. In particular, it the LLMs were to install a known chess engine and run that to get the moves when asked to play chess, that would qualify for more AGI-like behavior. But really, chess is such a simplistic game that they should be able to do decently well at it even without even needing that. At the very least, they should be able to consistently play without making illegal moves - something that many 7-year olds manage quite well.
If something has general intelligence it should be able to read the rules of a game and follow them. Therefore an artificial general intelligence (AGI) should be able to do this.
So we have a situation where very powerful and influential people are saying we will have AGI in 6 months (if we don’t already), yet the facts on the ground are so clearly pointing in the opposite direction.
I would bet a lot of money that Astra can follow the rules of chess (perhaps if repeated within the context window). Also, this is a different argument than what I responded to.
I can write you a benchmark to prove it even with a heavy handed system prompt Astra will make an illegal move during the course of the games first few moves are generally ok since it's just throwing out learned moves.
I wonder if I would do better as a human, maybe? Or would I happen to have one move in 1500+ that's not valid?
I could see myself messing up something at some point if the board is complicated enough and trying an illegal move, perhaps if a piece somewhere would attack my king if I moved another piece. Even through I do know the rules of chess, and I have played a few games once every so often.
> generally read the rules of a game and then follow them
How many times do you think chess.com prevents illegal moves from being executed? Even Super GM's fall for mate-in-1's occasionally, which is functionally equivalent to missing a pin or a check. This idea that LLMs failing to only ever make legal moves undermines their intelligence doesn't pass the smell test.
It only matters if you are claiming it to be general purpose.
If you admit that it's just a collection of narrow capabilities - whose strength is mostly confined to the 1000 or so RL environments it was post-trained in, then there is of course no expectation of it being general purpose.
The AI companies seem to heavily want you to believe it is some some near human level general intelligence, so therefore pointing out all the things it can't do is very relevant.
Because we want to use this as a replacement for humans, and the average human can learn the rules of chess without needing to see the rules explained hundreds of thousands of times in millions of games.
So, yeah, it matters if a model has millions of examples of something in its training set and still cannot follow the rules.
We're not talking about learning the rules of chess here, but playing a competent game from just being shown the rules. Why is it so hard for people to keep track of the thread of discussion?
> We're not talking about learning the rules of chess here, but playing a competent game from just being shown the rules.
Okay, lets go with that: it's the "shown the rules" bit that we are arguing about.
The argument is that a human may play maybe a dozen games after learning the rules, after which they won't be inadvertently attempting illegal moves. What we are observing with SOTA models is that, even after seeing millions of chess rules, rulebooks, actual games, etc, they still attempt illegal moves.
This does not point to generalisable and adaptable intelligence, such as we see in the average human.
This is not good reasoning. Humans need at least dozens if not hundreds of reinforcement sessions to only make legal moves, and still occasionally fail (consider pins, discovered check, failing to respond to check). LLMs must one-shot a competent game after imbibing a mass of disconnected units of information about chess. Nothing about the two are similar.
It would be more impressive if they could play chess (or do anything they haven't been custom RLVR trained for) by reasoning, rather than just "have a go at it" prediction which is closer to memorization.
HOW you do it makes a big difference in how you should assess the capability of the thing doing it. Stockfish will trounce any LLM, and any human, at chess, so should we say that Stockfish is smarter than both?
Yes, a good analogy. Except the cat actually follows the football rules and can beat some humans. And has no physical limitations to play other kinds of sport that you might imply.
You are saying "No it is not" without an argument. The fact that computer systems could play chess yet not being AGI has no relevance to LLMs' ability to play chess being AGI, because the point is about G, not I. There's little doubt about A or I parts.
I'm stating that certain folks are trying to use the software-generating product as an AGI/ASI and then complaining when it doesn't play chess very well.
People are holding it wrong, deliberately or not. Some are inventing bad faith measures so they can claim AI sucks.
Then why respond at all for the sake of responding?
We all know AI can code, but the question it all stemmed from what if it's AGI or GM level in chess on it's own.
You can't just back pedal from the statement that apparently being able to code a chess engine is the same as being good at chess.
I can write a chess engine that beats Magnus Carlson without AI that alone neither makes me GM level or AGI or any of the other claims the above comments seem to be making?
It's not even a "software-generating product". It's only half of it. Most of the heavy lifting is done by absolutely not-AI compilers, analyzers and the like. If not for these programs, written well before AI boom, them LLMs would be no better at programming than they are are at pure LLM based calculations or writing.
I agree w/ this perspective. An agent with a harness that can run programs can solve a lot more than one without the harness. The AI system includes the harness, and it's not clear to me that AGI requires more than LLMs + code generation & execution are capable of.
So AI is AGI in fields where code can't solve anything?
Is code omnipotent, I have been in software all my life and I would hard agree here.
Sure stuff LLMs can do with being good at parts of code reproduction is incredible. And honestly it's the new way to do a lot of things but I have not see an iota of proof that it can scale across the board.
For instance Maths is just code with different symbols and slightly less universally legible concepts.
AI is the best invention at figuring out or walking the search space and directionally doing logically computation over general software adjacent stuff.
But that's it, I am certain a bunch of companies will make a lot of money despite no AGI.
I think people either don't understand AGI or don't understand how real world works.
Until an LLM can bow it's head take responsibility for mistakes made and ensure they aren't repeated again with 100% confidence to the leadership it's inarguably a tool a rather questionable one at that.
> AI is the best invention at figuring out or walking the search space and directionally doing logically computation over general software adjacent stuff.
So.. like chess?
Anyway, do you have any prediction on what LLM's can or can't do in a few years?
First, you're moving the goalposts. Second, it's not actually true that any existing frontier AI can write a chess bot program that can beat a 1600 player ... not unless the program is derived from Stockfish or some other leading engine that has been in development for decades.
> The system is built and delivered as it is because that's how the providers make the most money. If they cared to have it perform well in chess games, you'd see a different shape and behavior.
These comments indicate a complete failure to understand the technology.
> I am around 1600 elo in over the board I can mop up Astra Fable etc even if I give them literal infinite time and all the subagents and internet access.
I don't believe this.
You refer to "subagents", so this is not just an LLM but an LLM with some kind of agentic harness. Any reasonable harness and prompt, given internet access and appropriately prompted to succeed on this task, is more than capable of firing up Lichess or chess.com and relaying moves back to you. The free levels will be enough to beat you.
A frontier model can also likely one shot a chess engine that plays at your level, again if given an environment in which it can do that.
I completely believe the LLM on its own can't play a full game of chess at your level. Though I'd bet that with enough reinforcement learning it is possible to train a pure transformer architecture to do that. We just don't do it because there are other approaches that play chess much better.
Even if you take that website at face value, the ELO scores shown are relative to the other AI models tested, and not comparable to the ELO scores of humans who play against other humans.
I wonder why they didn’t throw a real chess engine in there for a baseline. There are engines where you can set the elo in the settings, so it should possible to see these LLMs relative to a human 1500 rather than just relative to each other.
This is true, but I'm not sure it matters? I was poking around at the lichess database recently and those elo calibrated bots are
remarkably well calibrated, their rating variance sticks out like a sore thumb compared to human players even at similar game volumes. So it should still be a decent predictor of how good a human at that level is, even if the playstyle seems alien.
These ratings seems very wrong, i have beaten GPT Astra max thinking in chess and my rating is close to 1500. The ratings here seem more accurate: https://chessbenchllm.onrender.com/
GPT-6 almost never suggests an illegal move anymore while even Sol still did so time to time
Probably tells us that without labs explicitly training/tuning the models or designing the harness (with fast oracle) the LLMs aren't going to get good at those areas.
> The actual current frontier plays somewhere around GM level.... It's also worth noting that the very latest models (GPT-6 and Fable 5.1) actually play worse than their immediate predecessors
Sorry, but I am not buying that 5.6-Sol is that much better than 5.6-Luna, which can barely be coaxed to reach the midgame with legal moves and an apparent understanding of what the position is.
What happens when you ask those same frontier models to write a chess-playing program?
I feel like, this is a huge stumbling block that many people have. They'll give a model their data, and ask it questions. I vastly prefer letting the model understand the schema, and then writing functions or programs to answer those questions. I feel like I get way, way better answers. I can have it write unit tests for those functions. I can fuzz test those functions. I can look for data that doesn't fit the schema. I can process new data way faster (and with fewer tokens). I can repeatably get the same answers from the same inputs. I can check the code into a git repo and track changes to it over time. I can share the code with other people. I can review the code. I can improve the speed of the code and get the same answers. I can review the error accumulation, and improve it. I can decide how to handle anomalies, and encode those answers.
It's really neat to see what a frontier model can do itself. No doubt.
But "play chess by hand" is a frankly awful metric. It's kind of like asking someone to take a cube root of some arbitrary decimal, in their head, with no scratch paper.
Asking a model to "write a throwaway program to do X" is vastly more productive and reliable than asking "do X". Running code provides a feedback loop, the model can iteratively improve the solution instead of guess. Even if you don't read the code yourself, you have a reproducible, editable, and auditable artifact if you need it.
To go a bit off-track based on your final sentence: my high school physics teacher would do any numerical calculation that came up first in his head, as an estimate, and only then use a calculator or write on the board. Usually the estimate was within ±10-20% of the correct value even for long combinations of numbers with a bunch of decimals. Cube roots didn't come up, but square roots did.
The point of that was to show the use of approximations and of having an idea how much a result should be, to guard against calculator typos and the like. I think that has some metaphorical relevance for the chess example.
It's a great test of cognitive abilities. There are many claims that current LLMs surpass humans in cognitive abilities so it's noteworthy that they underperform on that test.
Letting the model execute a chess program (that it wrote) would make sense if you're measuring its economic potential, but for cognition that would be cheating just like if you let a human run a chess program. The fact that the human would have a much harder time writing a useful program is irrelevant.
> The fact that the human would have a much harder time writing a useful program is irrelevant.
Why?
There's a box.
You give it a problem, and it comes up with a solution.
Why does it matter to you if the box is strictly a LLM, or if the LLM can write code that it executes?
Even neater if the box is self-contained with a local model. You provide electricity, and it comes up with solutions. Why does it matter if it can do chess "in its head", or if it has to use scratch paper?
The question is to what end? This is a benchmark task, because playing chess, or solving other well-understood problems is more of a party trick than it is useful.
If you let the LLM write a chess program, which it can ONLY do because there are already so many chess programs out there, then the benchmark becomes about recall of popular program source code, not chess.
Neither. As I said I want to measure cognitive abilities.
Your "ability of the box" is like "economic potential" in my previous comment. If that's what you want to measure, fine. But I want a deeper understanding: what is the thing doing, how is it solving problems? I want to get a sense of its abilities that is richer than a one-dimensional scale.
I agree that it's a fascinating to crawl inside an LLM, and also to crawl inside of a human, and try to understand the processes and limitations. Like, Phineas Gage is one of the most remarkable learning opportunities we ever had.
That said, it's really weird to me when people use (and judge) LLMs one way... and won't try using them another way.
Like, to judge their utility, I think we should be open to letting them write code, and use the code they produce.
Otherwise, it's like judging a Chromebook without an internet connection. Like, this was one of the most dishonest ads I've ever seen: https://www.youtube.com/watch?v=gDy9AUQJ3Fg
This lamp, without a working power outlet? It really doesn't do anything...
>> There are many claims that current LLMs surpass humans in cognitive abilities
Where? By whom? This is certainly not (yet) the general consensus, as I understand it. Are you taking the most optimistic / untethered comments as the strawman against which you feel the need to argue?
Why the aggressive tone and the strawman rhetoric? I never said there was a consensus. Yes I'm talking more about the "optimistic" commenters and pointing out that this chess thing is a good datum to temper their enthusiasm. What's wrong with that?
Also these claims are not completely without merit, it's just that LLMs seem to excel at specific "cognitive" tasks and it's interesting to see where they fail.
It isn't. Stockfish running on your laptop can beat every human being on earth easily at chess. It's not intelligent _at all_ in any sense that matters.
Well it's not a perfect test so you need a bit of care in how you use it. If you have no idea what the subject is doing, then you don't know if you're measuring cognitive ability or something else (like cheating ability, or algorithmic sophistication or whatever). But failing the test is a pretty clear sign of certain cognitive abilities being poor.
I suspect (in a probably ignorant fashion) that this is because learning process has been reading a lot of algebraic chess notation (such as "1. e4 e5 2. Nf3 f6 3. Nxf6 gxf6 4. Qh5! +-") then, to play, generating more of it without considering the rules of the game. This is exactly how it's always felt to me when playing chess against LLMs. Sure, "1. e4 e5 2. Nf3 Nc3" looks innocent to somebody simply learning the syntax of algebraic notation, but that Nc3 by black is an illegal move.
Are you saying that modern LLMs cannot play chess now, or that LLMs (GPT architecture) cannot be trained to play chess well?
Or are you saying that neural networks in general cannot (practically) be trained to be an above-average chess player?
Or are you saying that it depends on the input? Would it be better if they were given a picture/drawing/ascii art of the board? If so, surely they can produce it at will?
Neural Nets can be trained to play chess very well and have been doing so for a long time (see Stockfish and Leela as some of the most popular/strongest ones - top GMs have no chance against them), but these are dedicated models, where the game rules are encoded in the learning process, as opposed to large language models which are natural language processing models. Technically you can give an LLM a lot of chess books and games and it will be able to spit out chess notation. Put a webapp on top that renders text moves to the board and it looks like it's playing chess. But it isn't really.
I suppose if you all you need is a good enough opponent for the average person out there, sure this is good enough.
I was more talking in reference to why the LLMs in the above linked paper were producing so many illegal moves, and it is because they are not hard constrained by the rules of the game. Of course, a loop can prompt until a valid move is produced and then rendered on a screen. But why do this? I suppose, who am I to say what should be done or not, but a specialized tool being better than a general one at its specific job isn't particularly surprising.
Yes "oversight and guardrails" are still needed, but even that is becoming easier to build, and imo already no longer in the "laborious" category. Even far from the frontier, you can tune a 0.2B LLM into a decent 2000 Elo player as a weekend project:
1. It’s hard to trust a 2026 paper that’s showing results for such old models.
2. Chess seems to be a poor benchmark for generalized strategic reasoning. People who are good at it rely more on experience and deep domain expertise than on skills that generalize to make them experts at unrelated tasks.
3. The study sounds like proving humans will never fly because they don’t have wings. In reality, humans do fly, and Claude Fable would destroy any human at chess by coding a strong enough engine on the fly.
Not sure how that vague truism applies to this paper.
Lots of papers have great results that don’t depend on the latest models.
However in this case it’s problematic:
- They specifically make claims about the state of “current LLMs”. o3 is not representative of this.
- They ask are LLMs capable of X and arrive at a negative result.
If their claim was LLM’s can write coherent sentences, and their conclusion was positive, then there would be no issue using old models because the end result would be factual.
However, when you have a negative result that makes a claim about the current state of all LLMs and the ones you were using are not current, by definition it draws the whole conclusion into question.
Maybe practically it doesn’t matter? Perhaps AGI is not the model but the model plus everything it’s got access to. If we’re modelling intelligence in the way we seem to have to to have any coherent definition of AGI, it seems to me <model + everything it can access> is always going to be more “intelligent” than <model> alone.
My points is more that, while we have a strong intuition about where, as an entity, a human's boundaries are (i.e. where the person begins and ends), philosophically it' not immediately obvious that the analogy applies to the a model in the same way. Why should that be the line drawn that says this is the "thing" and this other stuff is external to the thing? It feels somewhat arbitrary.
Of course this is a difficult question with humans too, hence my reliance on intuition above. We don't have the same cultural/biological framework to fall back on with AI.
I think all this debate about whether an LLM can write (or download) a chess engine is sort of missing the point. For basically any economically valuable work there is no equivalent of a chess engine for it. If there were we wouldn't need humans or AI to begin with.
If the goal is merely to "win at chess", then yes, an LLM using stockfish is better than any human alone at performing the task. When you are talking about what AI agents are capable of doing, there is no such thing as "cheating". They are as capable as the tools they can use effectively. The entire history of human civilization was driven by effectively using tools to achieve goals.
so prove it! get a public repo out there, have it play against some open source engines
also I think the operative letter in AGI is the G - and if the G is short for 'variably competent savant-like hyperfocus on certain kinds of software coding and not any other general skill' then its not really G at all, is it?
Declarative knowledge is not the same as procedural knowledge. You can read as many chess tutorials, strategy documentation and game archives as you like, they won't make you good at chess until you actually start practicing chess.
The issue with the models isn't that they play a bad game, but that they persist in making illegal moves. An average intelligent human can be told the rules of chess and then play chess, badly, within the rules.
Sure, but the LLM is free to construct a representation of the chess board and update it as it goes along. It is not in any way banned from using a virtual board, or whatever representation of game state it pleases.
AFAIK, current models will still sometimes make illegal moves even if given the entire game state (e.g. in FEN notation), so it is not purely an issue with the models’ ability to keep track of sequences of moves.
My point was you are misunderstanding G, or at least applying it erroneously here. Being good at chess is not a generalization of any other body of knowledge, it is a rigorous set of rules. The only way to be good at chess is to practice chess, or to apply deep calculations. The latter is the model writing code.
The illegal move aspect has more to do with a failure of online/in-context learning, which would support your point. I tend to think it is a byproduct of reasoning in language, which newer architectures would fix, but we shall see.
chess is not just a rigorous set of rules, it is rules as foundation with layers of strategy on top. and so is, for example, scientific methodology or chemical interactions or virtually everything else under-the-sun that comprises human knowledge
knowledge for chess is derived from memorizing strategies that have been well-defined for decades paired with in-game reasoning processes. this is not at all different from any other body of knowledge. Noble gases, laws of thermodynamics, organic chemistry just to name a few - these are all 'strategies' that define observed phenomena, analytical frameworks that trace a logical, rational set of interactions and which can predict the next
for an AGI, all of this should be a cakewalk, trained as it were to surpass human capability in any and every domain [0] (thus the G for 'general' and not 'N' for 'narrow' [1]). it should be a natural at everything, infinitely adaptable on-the-fly. the whole point of AGI is that it surpasses human capabilities even at our frontiers and bleeding edge (unless you're private enterprise and you've moved the goalposts for industry [2])
currently, it's only AGI-seeming if it gets benchmaxxed enough. otherwise it sucks at what it does and then is only barely competent at tasks if paired with enough skills and tests to make it more diligent at its work. this makes sense to me - for any probabilistically trained tool, even one that you post-train and fill with nothing but the best-quality evidence, the ultimate result is the lowest-common-denominator output for your sample set. there's no natural reasoning the AI does itself to make itself better at what it does - it's all human curation and categorization of sources ingested paired with RLHF post-training that we can get the mediocre-at-chess-at-best results that we see now and the benchmaxxed scores against whatever arbitrary and pre-defined measure
that's not AGI by any classical definition. that's a cool, useful, and powerful tool that makes our lives easier, much like a hammer, nail, and studs make mounting a picture frame easier than if we only had our hands alone
I'm not. AGI is almost necessarily closer to ASI than it is to human intelligence by definition. it's become pretty obvious that even what seems like irrelevant domain knowledge has utility applied to other domains - that's why we're pursuing general-use models
presumably, an 'AGI' that is generally as good as a really good human at every task under-the-sun will already be much better than most humans at the task because it can incorporate cross-domain knowledge and apply it in a reasonable fashion. it's like the parable of Newton and the apple - the domain knowledge that an apple falls according to certain rules observed through historic experience igniting the creative spark that led to universal gravitation
> presumably, an 'AGI' that is generally as good as a really good human at every task under-the-sun will already be much better than most humans at the task because it can incorporate cross-domain knowledge and apply it in a reasonable fashion.
I disagree with this definition of AGI, and I disagree that chess skills significantly benefit from generalizing non-chess knowledge, outside of computing moves probabilistically.
AGI has historically been defined as human level or better, with generality to new domains. I think blurring it with ASI makes the terminology confusing to use.
Chess is learned rules and the ability to apply those rules. Strategy as a whole is applying a set of rules to circumstances, that's how it is taught: "here are examples of circumstances and actions, try to pattern match to future circumstance and apply commensurate action."
If you make the point that chess is a large part of the training data, or that LLMs are unable to learn chess well, I'll accept that as refuting that LLMs are AGI, but these other points I disagree with.
> People who are good at it rely more on experience and deep domain expertise
People are good are 1900 or 2100 above and the top ones who spend decades in the field i.e. deep expertise are well in the 2200-2700 range.
A 1100 player is none of these things, they are purely relying on strategic reasoning there is a good chance they cannot name a single opening or articulate clearly why a move was appropriate. 1100 is quite low bar.
1100 at online speed chess or something, could be. I'm not that deep in the chess world but everyone I know that can make 1100 in official rating can name a dozen openings and most of the known tactics, and is pretty good at applying at least one opening.
1100 lichess/chess.com does not represent real elo. I'm around 1400 online, I would still be unranked in the real world. The fact that I easily beat any model publicly available is not a great look for AGI.
If the goal for buyers of AI is “replace this knowledge worker”, how much does it matter that the model in a simple loop can’t do it, but the model with a strong general purpose harness and a little time to gather resources and knowledge to augment the harness going forward, plus tool calls, plus custom built tools, etc, can replace the knowledge worker?
Probably the only thing saving many jobs from being replaced right now is that it’s hard to have a verification of correctness in the loop, so the agent can’t hill climb very easily.
people on the ground know that small models are enough, since llms are good at directed work (i.e handholding) not the let loose go wild that the labs try to hype on.
the only thing that few people are willing to admit is that humans are the bottleneck as humans are needed to handhold / verify output - which puts a dent or might I say pause on the excessive valuations of a.i companies as that's against the narrative.
By the promise of it, llms should be able to both adhere to grammars, or go free form where necessary. I mean, doing math is supposed to be strict but in practice it's a somewhat educated random walk in the space of correct lean theorems.
If you give the same task to an exceptionally intelligent human, who does not play chess and has only heard about it in passing, then they would be beaten by every child who has looked at the rules for more than 10 minutes.
What kind of intelligence is "playing <____> but we don't tell you the rules" supposed to test?
...but that's not what the models are. You can interrogate them on the rules of chess, and they'll (statistically likely) give you a decent breakdown of the rules. Evidently the rules are in their training material, they just fail to apply them in the manner of an intelligent system for some reason or another.
With that approach, the benchmark falls apart. Of course it can write a chess engine, because it learned on lots of stolen source code of chess engines. This has nothing to do with the LLM's ability to reason.
Writing a well understood engine for a super popular problem does not count as reasoning about the problem.
In most humans, yes, because we are terrible at memorizing millions of codebases. For LLMs, we need to apply our understanding of them before making statements like that. An LLM can "memorize", and has "memorized"/been trained on tens of thousands of chess engines. Writing a chess engine, or even deriving a chess engine from the rules alone, does not constitute a deep understanding of, and more importantly, the ability to apply, the rules, at all.
When humans do this, they inadvertently learn something, too, but when an LLM reproduces or derives and implementation of a chess engine, it in no way implies that the LLM can follow the rules in its own "train of thought" and consistently apply the rules in its "head".
Let's say you want to evaluate my algebra skills. You make me solve some algebra challenges. If I then whip out a computer and write a calculator, or take some sticks and stones and take a couple hours to build an abacus, and then solve the algebraic challenges, this would not constitute a good solution, and would defeat the entire point of the test. If, instead, I do the algebra in my head or on paper, it might seem like there's no difference, but you can derive all sorts of information from that.
For example, you could time it, check for recurring errors I make, for interesting mistakes like mistaking 7 and 1 for one another due to bad hand-writing, etc.
If that was the goal, then me writing a calculator or crafting an abacus defeats the point of the test. Yes, me writing a calculator shows that I'm intelligent, and I understand the algebraic rules, but if the test is about applying the rules, I have not passed.
In the very same way, an LLM writing a chess engine to solve a chess benchmark that is all about LLM's reasoning capability is complete bogus and defeats the entire point.
>Tool use is a widely accepted measure of intelligence.
Stop anthropomorphizing the parrot. The parrot has had stockfish's source code blasted at high pressure into its head along with dozens of millions of other pieces of code whose sole role is to have efficient algorithms to more or less brute force through the best result. Brute forcing (no matter how smart it is) isn't understanding the problem domain.
I'm not fully convinced that human beings aren't stochastic parrots. Much of the behaviour I observe in daily life reflects a blind adherence to set of beliefs that are an amalgation of "a person of authority said to do this" at an early age.
It all depends on what prompt you use though. You can just tell all current frontier models to write a chess engine first, and then play a game of chess against you using that engine.
It will probably do a pretty good job if you ask it that way (it will also burn a shit ton of tokens, but hey, that is part of the fun).
On that note, I actually had an overall harness (for experimenting) that was essentially like this: "for any task, instead of answering question directly, write a program to answer the question instead. test and verify the program before giving the answer".
It actually worked incredibly well on all "gotcha" LLM questions like math or counting letters in words and all sorts of stuff.
Of course it was ridiculously slow and very expensive but it was a proof of concept that it can actually be much more accurate on every task if you are willing to spend an infinite amount of money.
This is an absolute nonsense. Any frontier model can implement chess program from scratch - modeling the board, checking legality, etc. If you asked e.g. GPT-6 to get good at chess and gave it a computer, it will get good at chess. That's an actual strategic skill.
Asking GPT to play chess directly using its reasoning only tests its reasoning ability to model chess state. Which it really is NOT optimized for.
This is also true for humans - people who don't have years of chess training can't really tell which moves are legal given an algebraic notation transcript. These people might have good strategic skills in different areas. Chess is just a very, very specific skill
It's an interesting puzzle, isn't it. On the one hand, the AIs are no good at playing Chess.
However, on the other hand, if you ask an AI to win a game of chess it has all the tools on hand to compete at the same level as Stockfish - it can re-implement an engine and even probably has a GPU on hand to train its own neural nets.
So should we say that the AI can play chess well, or that it cannot?
I want you to consider how relevant this is in any practical sense.
First -- most _people_ cannot do this, without having a physical board in front of them.
Second -- Claude Code is perfectly capable of downloading and running stockfish. People focus too much on LLMs by themselves as the entity of concern instead of the entire harness and all of it's capabilities together.
Well, yes, PGN files have structure... But still, playing Chess with an LLM is so weird that I impulsively question the sanity of people attempting to do so. Do some people really believe training on TWIC PGNs would make an LLM a good chess player?
> Researchers asked frontier models to play chess. Have a look at the MAR rates in Table 3. When not explicitly told which moves were legal, no model identified legal moves at a rate better than 80%. Many asked for more illegal moves than legal moves. And even when explicitly told which moves were legal, the models continued to ask for illegal moves. With illegal asks discarded, none of the bots could beat a chess model calibrated to 1100 ELO.
It can be very interesting and even entertaining to know where models don't do well. I don't find something like chess to be very instructive about anything though, nobody is going to pay for AI to play chess at any significant scale even if it could do it perfectly.
> The author of the originating post says that "current frontier models need laborious oversight and guardrails on even the simplest tasks", and he's absolutely correct.
I have found that not to be the case, and I am not an AI power user or cheerleader by any means. There's probably a bunch of even "simplest" tasks where AI doesn't do well and might never. That doesn't take away from the cases where it works well and is a productivity booster. It doesn't even have to be solving millennium prizes or any other breakthrough creativity or reasearch, it's still very useful in places.
Frontier labs don't care about chess. If OpenAI cared, GPT-7 could be a grandmaster+ level chess player. In fact there's a google paper on grandmaster level chess without search with a 270M transformer. Outside that, there was gpt-3.5-turbo instruct which was incidentally a 1800 lichess elo player that didn't make any illegal moves even after a few thousand moves.
Frontier labs care deeply about automating knowledge work and computer use. They are working hard on getting models better and better, and they are succeeding. Astra is a step change on that front. So good luck i guess, if chess performance is your barometer.
> Frontier labs don't care about chess. If OpenAI cared, GPT-7 could be a grandmaster+ level chess player.
If the models were actually intelligent, the way that the boosters claim, they wouldn't need to be tuned to play chess in order to be good at it. That's kind of the point of intelligence, that it is generically applicable to whichever task one wishes.
Humans don't need a lot of training and finite tuning to make only legal moves.
An intelligent adult could simply read a short summary of the rules of chess and then, if they were careful, play a very bad game of chess without making illegal moves.
An LLM that has not been trained on any chess data cannot do that, at present. If you doubt it, take a current model and tell it that you want to play it at a variant of chess where, say, knights can also move diagonally like bishops. A human can easily adapt to this new ruleset (even if they make tactical mistakes, not having practiced with this variant of the rules).
How long a prompt do you think would be required to cajole an LLM into making legal moves at the rate of a human? Or do you think no amount of prompting could do that?
I don't know. My understanding is that current models will eventually fall into making illegal moves in longer chess games, and that no amount of prompting reliably gets them to stop doing so.
More importantly, beginner human players don't exhibit that tendency. The history of the position doesn't bother a human (except as required for castling and en passant rules), and the analysis becomes generally easier as pieces come off the board.
Nothing is forcing the LLM to play 'blind'. If it's smart, it should be able to create its own representation of the chess board and update it with every move, just like a human would. Any chess engine that's sensitive to how the moves are formatted is clearly not very capable.
A human wouldn't do that, they'd look at the board. I'm not disagreeing that to demonstrate clear superhuman ability the LLM should be able to do this, but it plays better than most humans blindfolded, and with fair prompts seems very good otherwise.
That's what a human will do if they already have a physical board to look at. But if someone, say, posed you a chess exam question via FEN notation, or as a sequence of moves in algebraic notation, you'd sketch a visual representation of the board off your own initiative to help you answer the question. There is nothing in principle to stop the LLM creating its own board representations in whatever format enables it to easily keep track of game state and legal and illegal moves. If it fails to do so, that's a sign of its own limited understanding of chess as compared to a human.
The LLM would only be playing 'blindfolded' if you somehow forbade it from making notes (as you effectively do by literally blindfolding a human, given how limited human working memory is). But you are not doing that. The LLM is free to keep track of the game state via whatever means it chooses.
None of this is about superhuman ability. Any human who understands a given chess notation can convert it to a visual representation of a chess board and then use that representation to choose their next move, with their usual level of performance.
I maintain that the amount of effort to teach a human to do this vastly outweighs the amount of effort to teach an LLM to do this unless you're deliberately trying to make them fail. I honestly have no bigger point than that, I just think this isn't a very good thing by which to evaluate LLM capabilities. If there's no argument you'll accept, I am happy to move on.
You don’t need to teach a human anything except the rules of chess and the details of a particular chess notation. No special skill or training is required to make a sketch of a chess board. Surely there is no chess player who, if confronted with a sequence of chess moves in algebraic notation, would not think to construct a representation of the chess board in order to understand what was going on.
> I just think this isn't a very good thing by which to evaluate LLM capabilities
I don’t think any single task is a good way to evaluate LLM capabilities, but I don’t see why chess is worse than a lot of other tasks. (Of course it is of no practical consequence whether LLMs can play chess, so if you are just making that point, then yes, I agree.)
> If there's no argument you'll accept
It’s a little unfair to suggest that I wouldn’t accept any argument whatever for your position just because I haven’t been convinced by your very brief comments so far. I could equally well say the same thing to you!
Pretty much this. Feed it a book or two on chess, and you should have a decent (or good) player. That's the generic intelligence people have. The aims is not to be supremely talented at something, but being able to read a manual and figure how to use/play something. Mastery can be gained overtime.
If you gave a human a book or two on chess they would not become a decent player (they would be closer to 500-600 than 1100 ELO) and they would only get better after playing hundreds or thousands of games (often making illegal moves and moves that violate the rules of chess as they learn).
Your assumptions/intuition about generic human intelligence feels quite incorrect, considering LLMs currently play better than a brand new human player would (presumably without any attempt to fine tune them specific on chess, such as playing thousands of games).
the discussion isn’t really about whether language models can become strong chess players though, the point is they seem to struggle to consistently make valid moves. Most humans don’t need to read two books to pick that up, just a couple lines of basic instructions
That has not been my experience with new players, they regularly make invalid or incorrect moves even after detailed instructions especially in novel situations.
Maybe it depends on the person? My six year old isn’t great at strategy but they can pretty consistently make valid moves. Sometimes they ask for confirmation on a move which is also not a trait I see in language models (at least unprompted)
That's quite untrue. I taught my (adult) brother the moves, the only illegal move he ever tried against me (over his 6 first games) was a castle with a rook that already moved twice. Within a few hundred games (less than 500 for sure, he played 3 minutes blitz but always took at least 10 minutes analyzing his games) he was rated 1100 on lichess (which is like 1050 on chess.com and unranked in the real world).
So your brother tried to make illegal moves while learning the game and it took your brother hundreds of games to get to be a decent player? I don't see how this contradicts anything I said...
The _only_ illegal move a human might make as a beginner is a failed en passant or a bad castle. And yes, a few hundred games is all it takes to be better than any publicly available LLM at the moment.
Yes, but my point is that humans can’t even do the thing that the above comments are claiming humans can do (read a book or two and be decent at chess), and then they complain that LLMs can’t do the same thing (that humans can’t do either).
We seem to be moving goalposts to the point that humans don’t even live up to the expectations of the AI critics. The only way you get better at chess is by playing a lot of games and learning from mistakes, that goes for humans or AI agents, not simply by reading about chess.
> The only way you get better at chess is by playing a lot of games and learning from mistakes
How can you play without being aware of the rules and how can you learn from your mistakes without knowing they are mistakes? That’s what I said about reading a book of two. It is to kickstart the process. Then mastery is gained over time through practice.
This kickstarting then gradual refinement is how most people learn. And the foundational knowledge stays. Even a basic player knows to not do illegal moves.
Reading can kickstart the process, but you can also make random moves guided by some sort of system (such as a computer GUI) or learn by watching other players play. The overall point is that you learn through observation and lots of trial and error (whether you are a human or a computer). And beginners in chess often make illegal moves even after learning the rules, it's fairly common.
It feels like you're trying to say that humans never make illegal moves while learning chess, which doesn't match with my experience. I'm trying to understand your overall point.
> The overall point is that you learn through observation and lots of trial and error
That’s the most inefficient way and people usually avoid doing that. Instead they find someone that knows how to do the thing and ask him to be a teacher. Or use a proxy like a book or videos.
> It feels like you're trying to say that humans never make illegal moves while learning chess, which doesn't match with my experience. I'm trying to understand your overall point
There’s learning the basic stuff (which is done after a few games) and there’s mastery. The thread started with the observation that even with all that knowledge (through content ingested in training), LLMs still makes illegal moves. Humans can be erratic, but they can constrain themselves to the rules for the task at hand after learning them.
The only way to master anything is lots of trial and error. Coaches and teachers can help guide you towards more focused trial and error paths but the student still has to do the lessons and put in the work of learning, and learning only truly happens through doing.
Humans are not perfect and make mistakes in learning even when they have memorized the rules. A simple example is new players will often move a piece, exposing their king to check, and a more experienced player must point out to them that they have made an illegal move (because a new player often has not encoded that pattern for looking for exposed checks because they're more focused on how the pieces move, not what that piece exposes.)
We're just going to have to agree to disagree here.
If humans were actually intelligent, they wouldn't need to train and practice to play good chess. I mean, what level do you think people without any practice or training are ?
OpenAI making the next model good at chess is not analogous to a human training to get good at chess. It is analogous to God creating Human 2.0 which now has increased chess playing ability. If LLMs were intelligent the way humans are, then the models that exist right now would be able to spend time improving themselves at chess and become good at it. They can't do this because they are not, in fact, intelligent.
Oh right. But if all you do is reading programming books you are an amazing programmer? Where is all the training and practice LLMs did to become so good at coding?
It's called post-training, typically through some form of reinforcement learning, and is a significant part of modern LLM development.
You have the first stage, pre-training, which is learning from next token prediction. That's where the model memorises a lot of facts about things and generally gets good at forms of writing. It's like reading a lot of books on programming and reading through a lot of source code. It's learning how to autocomplete code, essentially. Doing that requires a developing a reasonable understanding of code, but it's also learning how to autocomplete bad code as well as good, and won't make it a "good" programmer.
Pre-training uses a method called Cross-Entropy Loss to update the weights of the network.
Then comes post-training. This is where the model is trained against huge sets of example problems, like fixing a bug, adding a new feature based on a spec, etc. They are set the task and try to complete it inside a training environment. Once they're done, their complete solution is evaluated (either by humans, or by some separate evaluation model that was developed based on human feedback) and they are updated based on whether the solution was good or not.
Post-training uses a different method called Proximal policy optimization to update the weights of the network.
So these really are very different forms of learning, and mainstream LLMs are not post-trained to be good at chess. They could be. You could easily create a reinforcement learning environment that evaluated and improved their ability to play and win at chess. The result would be a very strong chess playing AI, something we know is possible because the strongest chess playing programs we have are neural network based, but it is not a priority for AI companies.
>Where is all the training and practice LLMs did to become so good at coding?
Coding is a matter of translating the natural language description of a problem to the code specification while keeping the semantics fixed (and imputing the unspecified semantics as necessary). It is not considerably more difficult than translating between two dissimilar natural languages. Chess isn't a matter of language translation, but a compute heavy game of finding the best move out of many possibilities with wide variation in the quality of each move. Chess takes directed practice and reinforcement whereas language translation does not.
LLMs (and Humans) don't get really good from programming books lol. The training and practice is the actual code they predict and learn from in the process of predicting.
Oh I see. So if someone just reads books AND actual code then they can become experts, got it. And by the way LLMs are also trained with probably hundreds of thousands of actual games not just books
This isn't how intelligence works. The LLM may not be able to play chess directly through inference, but it can write a program to do it and execute that program. Same as how human intelligence works. We can't fly, but we can build planes.
Poorly in what sense? I think human chess leagues are way more popular and fun than just playing a computer by yourself. Human oriented communities are always a vastly better experience than their digital counterparts.
There's more to games than simply winning you know.
And how many years of direct play and study does it take for a human to get good at chess or any other game? Absolutely no human ever could be good at chess just by reading a few books, or even every book on chess. That's just not how the brain works. If LLMs could do that they would truly be superintelligence.
> Absolutely no human ever could be good at chess just by reading a few books, or even every book on chess.
Maybe not, but you'd be surprised how little it takes.
A six year old child can learn the rules of chess well enough to be able to play legal moves only in a single day. And they can improve their game at a pace which is almost frightening to behold. I have taught children, and I've witnessed significant improvement materialise in a single game. LLMs have probably thousands of chess books, games, videos, etc in their training data, yet they are unable to even follow the rules.
This is, at the very least, interesting. It illustrates many of the things brains can do, which current ML systems in general, and LLMs in particular, can't.
It is interesting, but people are drawing the wrong conclusion from it. For one, LLMs don't go through a "chess learning phase". They're not analyzing a board as they're learning the rules or studying games to create a coherent model of chess. They're just imbibing raw relationships as isolated fragments of information. The fact that they can't unify this into a coherent model of chess playing in one shot and execute a competent game says nothing interesting about the limits of their intelligence. If you give frontier models the rules of chess in their context window, could they perform only legal moves? I bet they could, excepting trickier scenarios like pins and failing to respond to a check. But those kinds of scenarios have to be reinforced in any human player as well. Even Super GMs fall for mate-in-1's occasionally which is functionally equivalent to those kinds of failures.
No, learning is definitely not a sign of super intelligence. I know words don’t mean anything anymore, but that is simply general intelligence, despite the claims we have reached this milestone.
We're obviously using different meanings for "good" here. But aside from that, it took 100's to 1000's of reinforcement iterations for your brother to play competently. While certainly impressive, that is still an entirely different category from piecing together disparate facts learned during training (LLMs aren't analyzing a board as they're learning the rules or ingesting PGN files), to executing a competent performance in one shot.
I think the line of criticism around LLMs sucking at chess makes more sense when you understand what the AI companies are saying about the future trajectory of these models.
The entire recursive self improvement story falls apart once you point out that there is not much "cross domain transfer learning". Meaning that training an LLM to become good at coding, math, etc, will eventually transfer into them being good at other skills that were not explicitly trained for.
Using games like chess which have little economic value is actually a good test for this. What's even more surprising about them sucking at chess is how much information about chess strategy exists in the training data.
There’s multiple databases of games in algebraic notation. You can also, very easily rl train on pitting models against one another, even without mcts.
It is even worse.. This is a classical reinforcement problem where data generation is easy because the rule set is pre-defined. So you really don't even need any data to start with (but would help).
There are more possible game combinations than atoms in the universe, even those generation of valid game states are as you say pre-defined. that is why models cannot go this route and therefore are poor at chess
Isn’t this exactly how AlphaZero was trained? The rules are known and well defined so the training process can generate games without any outside data.
The only reason LLMs are this bad at chess is because the labs don’t care about chess performance so they’re not going out of their way to train the models for it. The ability they do have is from what chess information happens to be in the training data, plus whatever general reasoning abilities they may be able to apply.
The last post on HN I read was about someone using LLMs to reverse engineer an Apple GPU driver for linux in a month. The top comment points out how the poster must have had specialist internal domain specific contact with Apple. But then the thread concludes that wasn't the case and that this would take domain experts years to do.
> "current frontier models need laborious oversight and guardrails on even the simplest tasks"
I feel this statement is extreme. I can't personally reconcile it with any of the projects we're regularly seeing get delivered largely by LLMs now.
What are you thoughts? Like, what's your position here? Even if you sincerely believe frontier models need laborious oversight on even the simplest of tasks, do you think that accurately captures and reflects the current state and progress of frontier LLMs?
Don't get me wrong, there's lots of things LLMs can't do well, but the idea that they're basically not helpful for even the simplest of tasks seems... disingenuous?
The fact that they can play chess at all despite having no specific training for it blows my mind, and the fact it doesn’t do the same for many others shows just how far they’ve come and how fast.
It doesn't blow anyone's mind because it hasn't been impressive for a computer to play chess for 40 years. "We made something worse than existing solutions by using a new technique" is not an impressive feat.
I don't see why this is such a big deal. Nobody's using LLMs for chess, but even if they are, just give them Stockfish as part of their harness. They don't need to do everything themselves as long as they're intelligent enough to use tools.
Navier-Stokes is a well defined problem, or "a hard technical problem". Most problems in the business world lack a good definition and tacit knowledge is required to solve them. As far as I've seen, AI lacks any kind of tacit knowledge, strategic thinking, etc what so ever.
Take a customer service person, that as soon as AI agents replaced was hacked. Lots of tacit knowledge, that wasn't measured, or even probably in the job description, until AI agents had none and the gap was taken advantage of. Gap is probably a poor word here as it implies not a chasm, which could very well be the case.
Your analysis was excellent but short on one front, AI has endurance on it's side. Looking at the N-S solution, OpenAI had 10,000+ instances that kept trying around the clock. Assembling a human team to do that would require a lot of effort. Though, that society collectively choose not to, perhaps tells how valuable it really is (i.e. it's now easy to launch a Manhattan Project level of effort). So maybe it can also be said that AI is also good at marshaling resources.
Great article, but the lack of sentence capitalization makes it unnecessarily difficult to read.
Apologies if this comment is off-topic, but it really is quite egregious, and since the article was submitted by the author I presume they are open to the feedback.
Hi! Thanks for the feedback. I've added an orthography toggle for those who prefer a more conventional look.
I will add that I'm not very happy with the readability of my site overall at the moment; if anyone has font or other recommendations for style tweaks to make I'd love to hear them!
To expand a bit: It's clear from the writing that the formatting is an intentional stylistic choice. My point wasn't about aesthetic preference. I meant precisely what I wrote. Language evolves and orthography evolves. We've been capitalizing less and less for centuries now. But (until very recently) it was a universal rule to capitalize the first letters of sentences. I believe this is partly because (again, until very recently) people read in large quantities and, to a fluent reader, sentence capitalization serves an important purpose: it helps the eye recognize where one sentence ends and the next begins. Periods alone can be easy to miss, or confuse for commas, when the eye is moving quickly.
Anyway, this is all a very minor point. You have a cool site and I enjoyed your article. I hope you keep writing.
> if anyone has font or other recommendations for style tweaks to make I'd love to hear them!
I'll play!
My recommendation, in short: pick a new font, make the content pane narrower (around 75 characters per line), and increase line height by 15%.
The font you use, Montserrat, has nice details, but they're extremely subtle, and our eye doesn't pick them up at text size.The font uses very pure geometry [1], has really big counters [2], is light in weight, and has no visible stroke contrast [3]. The effect when you look at the screen is that you see big blocks of text, but struggle to register individual characters, and it's hard to find the beginning of the next line. Add the lack of capitalization, and it feels like a monologue, like you're playing rubber-duck for someone.
A challenge: everybody uses Google Fonts, and all the good fonts get used so heavily they end up losing their ability to make people feel something.
Pick a font that is good [4] and feels comfortable to you. Ideally, buy one. You get what you pay for. And you go from being one of the millions of people who use a certain font to being one of fifteen.
If you like the feel of Montserrat, Tiny Grotesk [5] and Decimal [6] would both be great choices and are from excellent type designers. Or browse Matthew Butterick's font recommendations [4]. They're good. You're out $50, but it's yours.
If you're absolutely allergic to paying for fonts, DM Sans and Work Sans are Google Fonts, have a similar feel to Montserrat, and solve the above problems. But then you're a robot.
A final note: Montserrat is actually a nice headline font. The details come into focus at larger sizes and weights. So I'm going to open a can of worms: you could pick a _contrasting_ body font. Remember PT Serif from the [3] footnote? Try it on for size.
----
[1] By that, I mean things like: an "o" looks like a perfect circle rather than an oblique oval, and forms are strictly on 90º axes.
[2] Counters are the apertures of letters, the open center of an "o" being one.
[3] This is the contrast in "line" size. Think of it like the width of the line when you write with a chisel-tip marker: some lines end up thinner, some thicker. Look at the "e" on the font PT Serif - the vertical walls of the character are thicker, horizontal thinner. https://fonts.google.com/specimen/PT+Serif?categoryFilters=S...
why take the least charitable possible reading D; i've always read and intended it as inviting informality. i also don't find it harder to read at all (most people don't know some find it harder to read: i didn't)
I didn't take it any particular way besides maybe a hint of informality. Regardless it definitely threw off my reading in an unpleasant way.
Anecdata point for the poster maybe. I would read again.
there is a very big difference between starting an ephemeral instant message without capitalization, and lacking capitalization throughout a more "persistent" long-term medium
huh i didn't even notice, that's how i write all my blog posts too. it looks nicer to me and i don't have to bother checking for "proper" capitalization if everything's just lowercase anyway.
didn't realize people struggled to read text that way though, maybe i should change my writing style if this is a common pain point
I personally find it slower to read text without capitalization. It helps me visually chunk information from different sentences, in the same way that line breaks separate ideas into different paragraphs. Without it, it’s sort of like a wall of text that I have to read more carefully to parse (although a closer reading perhaps isn’t bad).
For what is worth, it is incredibly jarring to me as well (I am a middle millennial, not a boomer, and usually not prescriptivist about writing style). My knee-jerk reaction (I know, book by the cover...) is that it is probably not worth reading given that the writer did not think it is worth making it easy to read.
Counter-point: Doesn't make it hard to read at all, definitely not "egregious" as it's a very common style to find for informal writing all over the internet.
Author's own style is certainly refreshing and welcome over LLM slop that dominates most HN posts now.
> the frontier labs are priced according to the narrative that they have produced or will in the very near future produce a fully automated drop-in replacement for most knowledge workers...
Even assuming this is how the AI companies are being valued (they're not), the numbers are off.
The "value" of most knowledge workers -- based on what enterprises currently pay for them -- is $50 - 70 trillion annually. It's reasonable to assume that if AI drop-in-replaced all those knowledge workers, AI companies could credibly charge somewhere in that order of magnitude, because that's what the market is already bearing.
So if their hypothetical revenues are double-digit trillions and valuations are some multiple of that, the entire AI industry would be valued at double-digit trillions at the least.
Yet cumulatively the industry (the frontier labs + the SWAG estimate of the AI parts of all the other players) are valued at, say, ~6 - 7 trillion? Which seems like a fair approximation of how much knowledge work they can currently automate.
> The "value" of most knowledge workers -- based on what enterprises currently pay for them -- is $50 - 70 trillion annually.
What do you mean? The sum of ALL US salaries is $13.4 Trillion per year. According to google $65T is the sum of ALL salaries Globally (not just knowledge workers). It's not reasonable to assume AI is a drop-in-replacement for any job yet (perhaps bottom tier customer support from oversees?).
> So if their hypothetical revenues are double-digit trillions and valuations are some multiple of that
So you're sort of premising here than more than 16% or 1/6 of all the world's jobs get replaced by AI. Hopefully you can understand that's both not the current AI capability and also would be a terrible (unprecedented?) economic shock.
You are comparing company valuations to annualized revenue (as approximated by some fraction of total knowledge worker compensation). Valuations are (roughly) based on the sum of all discounted future cash flows, not just the current year’s revenue.
That's what the 30x instead of 20x multiple is, for a "growth" tech company.
There's no strong evidence that openAI or anthropic will have non-linear revenue growth, so I'm not sure what point you're trying to make. Unless you think they'll fire all their engineers and replace them with agents or something crazy?
It does not have to be 16% of all jobs, but 16% of any given job, i.e. AI stays in an augmentative role rather than a complete job automation. The simplistic analysis is if a tool makes you X% faster, that can be worth X% of your salary to your employer.
Unfortunately, I do fear that AI adoption will go beyond augmentation to automation, and I do fear an economic shock. Just posted this down-thread: https://news.ycombinator.com/item?id=49722616
You said 10s of trillions of dollars of revenue, which when the sum of all salaries is 65T, is minimum 15.3% of all salaries best case. It really feels like you're just pulling numbers out of nowhere here.
When thinking about these valuations, shouldn’t we try to quantify how much knowledge work becomes obsolete if other knowledge workers are automated? I.e. there are a huge amount of knowledge workers employed in businesses that create tools for other knowledge workers. AI won’t automate their work, those businesses will just cease to exist.
And then there’s the second order effect: if all the knowledge workers get automated, who is going to buy the stuff that’s produced?
I think you are committing the lump of labor fallacy [1]. Lots of jobs will disappear, but others will appear. Lots of things (both intellectual and material) that are produced nowadays by humans will be produced in the near future by AI. But humans will be needed to do new things.
Take the Hugging Face incident. Why did it happen? Because the people whose task was to set up a testing framework took shortcuts. Why did they? Because there weren't enough people who were assigned to do the job. Why not? Because the job is too new and not enough people are qualified to do it. It's a job that simply did not exist 3 years ago. But 3 years from now, this job might very well employ tens of thousands of high skill knowledge workers.
That’s fair. I guess I’m also thinking that whatever new kinds of intellectual work that arise from the introduction of AI could be also done by AI. But maybe this comes from a lack of imagination again.
Oh for sure, this was a simplistic analysis assuming AI adoption caps out at some X% of job responsibilities where X << 100%.
Unfortunately, I fear that may not be the most likely outcome. I've posted some comments on this before, but when I start thinking about how deeply everything will change once people figure out how to properly leverage AI, I see no outcome other than significant, widespread job losses.
As you indicated, at that point we will have much a bigger problem than the valuation of the AI industry. I'm not sure how it will get solved, I just know it will HAVE to be, because it would be an existential problem for everybody: people, governments, even the billionaires! Because now consider the 3rd order effects: if nobody can buy the stuff that's produced, how can billionaires get even richer? ;-)
The real issue IMO is that is not really what Anthropic and OpenAI are operating on.
That is the after the fact justification of the AGI dollar auction. Each round is kind of 3x the previous cost and neither can really stop because second place in the dollar auction is so much worse than winning.
The only way to stop the auction is one bidder hits a hard budget constraint, both agree to stop, or an outside party breaks the auction.
IMO this is why they want to slow down or have regulation. I think this is also why we see some claims of already reaching "AGI".
The TAM of global knowledge work is just a narrative tacked on after the fact to justify the AGI dollar auction.
The economic fallacy here with the actual valuation is akin to pricing the electric utilities 120+ years ago as some % of the future cash flow of global food production. Take the TAM of global food production and then work back to what % will the electric utilities capture from the advances in the automation of farming? It is nonsense.
The only narrative that actually justifies the capex spend that I can figure out is a first mover AGI monopoly. Even the oligopoly case is hard to justify the capex spend IMO. There is this enormous mismatch between the AGI monopoly and the actual rolling 12-month window of pricing power.
Even the rolling 12-month window of pricing power is going to saturate well before AGI too so it is hard to see how any of this makes economic sense.
> It's reasonable to assume that if AI drop-in-replaced all those knowledge workers, AI companies could credibly charge somewhere in that order of magnitude, because that's what the market is already bearing
This assumes you don't change the market, but at the scale of (checks notes...) "all knowledge work", that just doesn't hold.
For example if you put 1bn people out of work, you now need some sort of safety net to bail out much of that workforce, a truly unprecedented change. You also lose tens of trillions of dollars of tax revenue.
One solution might be to recoup that cost and lost tax revenue from businesses by raising corporation tax. If corporation tax went from low tens of percent to high tens of percent, would those businesses be able to afford all that AI? No. Same order of magnitude? I doubt it.
There are many possible futures there, but the simplification made in the parent comment is completely unrealistic. The article is right in calling out the valuations as crazy.
The changes that AI technology bring are unprecedented, it’s hard to predict what the impact of widespread adoption will look like. Probably collapse of price as a decentralized mechanism of ressource allocation, and a deep inevitable paradigm shift in economics and politics. It’s as if the actual value of those companies can’t be reliably pinned on any currency amount!
In one case (financial services) it's thought that expertise is valuable at V=S^2/b4 where V is value, S is skill and b capacity (the leverage available to the manager/expert. b erodes as it becomes harder to find examples of things that are not done well, so if you manage $1bn you might find lots of miss allocations that you can exploit with just that $1bn really effectively, but if you manage $10bn it's much harder to find good places for the extra $9bn. A low hanging fruit effect.
Anyway, that double hit - raw skill and the amount of times you can supply the skill makes the value of skill (V) convex, and it means that in a perfect market (heh heh heh) someone running $100bn is worth 1000's or maybe 10,000's of an average joe expert.
Now, if AI is trusted to run the top 0.1% of everything and has the skill to do it at human top level expertise, then your calc holds. If it's the case that it isn't then more than half of that value disappears. If it's not even top 1% then chop out another 25%.
That implies that we need a lot of trust and a lot of AI capability before these valuations stack up, and it also implies that all other competitors and incumbants are going away. I do not think that Citidal or Bridgewater are going to let Anthropic or OAI take them without a fight. They might lose - but there is a decent bet that they don't. I don't think that many professions like Lawyers or Doctors are just going to roll over and cede their monopoly rights to OAI or Anthropic either.
> The "value" of most knowledge workers -- based on what enterprises currently pay for them -- is $50 - 70 trillion annually. It's reasonable to assume that if AI drop-in-replaced all those knowledge workers, AI companies could credibly charge somewhere in that order of magnitude, because that's what the market is already bearing.
I don't think that AI companies can charge the same. The human workforce can charge these costs, because of scarcity. But AI systems won't be scarce, it's just a matter of who can run inference cheapest. Plus you still have the human workforce, which might be forced to offer their time for less money.
> It's reasonable to assume that if AI drop-in-replaced all those knowledge workers, AI companies could credibly charge somewhere in that order of magnitude, because that's what the market is already bearing.
Future supply and demand will set the price - not what is paid today. If supply by open models is vast and cheap, I can't see that the entire knowledge industry can hold the current size. It'll rather collapse to a fraction of its current value.
I would add to the other comments that are replying below the following thought : replacing workers with AI means less income/spending, which drives the value of those companies down at the same time.
You’re right; given that most of the money in the AI market is injected through OpenAI and Anthropic (which collect it through both selling equity and through customer revenue), the 7-8T is just a derivative of that.
"are priced according to the narrative that they have produced or will in the very near future produce a fully automated drop-in replacement for most knowledge workers, "
No, they're really not.
They're priced in a way that would imply AI will be universal form of compute, alongside traditional deterministic systems - which it will be.
And that they will capture most of that ... which they won't.
The Frontier Labs are a very bad buy at a high price, but that partly has to do with wacky pricing, but actually mostly has to do with their relatively weak place in the value chain.
The money is going to Nvidia, who have the most powerful position.
A bit like how a retailer can take all the margins of some innovative product, if they own the channel.
AI is over-hyped, the Frontier Labs are over priced - but AI is here to stay, and will grow. Not like Skynet, but like a new form of compute. And it will take it's time, and the profits will be reaped by those with the power.
I used that description some months ago and I think you are the first person I see who put it the same way.
The only gotcha with this is that they are theoretically deterministic, but rarely in practice.
A few examples:
- Harness specific settings that user can't control (anything from timestamp to prng seeding.
- Batching requests in a way that leads to a single request being processed different depending upon the batch (say MoE where your first choice expert is assigned to someone else's token so you go to your second choice vs a batch where you get your first choice).
- Graphics card itself carrying out floating point arithmetic in slightly different orders leading to floating point non associativity causing different outputs.
But all of these can be controlled for (at some cost) and the model can be ran deterministically.
For the average user, it might as well be non-deterministic, but when considering theoretical capabilities, chaotic deterministic system seems the better description.
> They are chaotic, which people confuse for non-deterministic.
That's an odd argument, because a lot of people who have struggled to decipher complex chaotic systems would tell you this is a distinction without much of a difference.
It's at least important from philosophical perspective regarding whether LLMs have free will or not. It's hard to say whether or not a human thought or action is driven by free will, people have debated it for centuries. We can provide the same input to an LLM and get the same output, because they are deterministic. Surely it is obvious that a pure function does not have free will, no matter how expensive it is to compute.
recalling the exact phrasing, several senior people at anthropic made public statements agreeing a 10% chance of causing extinction in less than 10 years.
10% is uninsurable, priced in with ordinary treatment of risk it suggests that anthropic should be worth zero today. creating that risk would put every executive in jail.
on top of that it would demand under existing laws of conflict, a military campaign to destroy anthropic. that is not optional, it is demanded now to save lives.
hard to make comparisons but we mourned and rembered 9/11 recently. a 10% risk of hundreds of millions dead in 10 years would make anthropic a thousands of times greater threat than al qaeda. many countries would assassinate dario amodei and the leadership of anthropic now, within weeks or months.
actually just on the vague risk of having a nuclear weapon in 10 years, the USA killed ayatollah khamenei, his daughter, his son-in-law, his daughter-in-law and his 14 month old granddaughter. then, they killed over 120 children ages 6-12 by accidentally bombing a school.
in the sense that i would analyse a company, at least, the claim is false. it's not true that ai has a 10% chance of causing human extinction within 10 years.
they are making false claims about the technology they sell.
i have a fairly inflexible approach to that. sure, exaggerate but outright lies about the nature of the product don't work for me.
I think bearish on LLMs for automation, and bullish for LLM+human experts in specific fields, is about the right expectation for current architectures.
Apart from issues with task generalization, or perhaps related to it, is the fact that LLMs have real trouble with timekeeping, and cannot estimate the real world time it will take them to do things very well. This plus the memory issues make dreams of long horizon agents, that could plausibly handle changing specifications, quite implausible with current architectures.
In narrow domains with more deterministic outputs though, this is less of an issue, and we see multiple agents succeed much better.
The fusion of that capacity, with humans in the loop able to better direct such agents and act as their temporal tethers, is where I think the real action will be for a while at least.
My belief is that LLMs will fundamentally change how we approach domain expertise. From what I've seen, SDEs tend to be over-specialized compared to what the company actually needs to implement due to the need to understand enough of the domain to pick a best path. If an LLM can see the domain enough so that someone in an adjacent field can be confident in their approach and quickly change course then you don't need as many niche SDEs
I think automation is coming but it will be way more gnarly than frontier labs want public to believe. Value is just too big, when you can automate most of eg customer support it will create huge savings and same time customer satisfaction will get better.
> This plus the memory issues make dreams of long horizon agents, that could plausibly handle changing specifications, quite implausible with current architectures.
Any reason why that can't be solved through context management and keep-forward scaffolding?
And who’s to manage context? And who’s building the scaffolding? Yes, AI can be used for both, but you do realize that all this being self-contained and regulated internally is what makes biological agents successful agents, right? If you break the process apart and need to dial back in these aspects, and can only do so with human input, or another agent which will need the same handholding the one whose issues you’re solving for, where’s the agency?
"Language Models are Few-Shot Learners" - 2020, the GPT-3 paper.
It have been demonstrated that in-context learning is a very powerful mechanism. There's no evidence that models of the size of GPT-6 are bad at in-context learning. In fact, ARC-AGI-3 score might indicate they are good at it.
There's no evidence that a bespoke RL environment is required for each new skill - quite likely a good demonstration is sufficient.
Short and to the point! Open and cheap models will undercut the big labs continuously. The blast radius won't be pretty once spending commitments knock the door.
Open models wont be open for long. No one is going to release an open model capable of chaining zero-days. Even the Chinese aren't that reckless because it will just be turned around and used against them.
As compute prices fall it gets easier and easier to make "frontier" models. So it's inevitable that commodity, open source models of equivalent capacity to today's "frontier" models will be available to the public. Remember this is just weights, anyone can download them and run it whenever they like. The only constraint is compute.
Isn't that assuming that fix won't be implemented?
Zero days are valuable because they can be exploited but if the pace of exploitation is faster (which I'm not sure is the case), then the response WILL be faster, even if it means going offline. Institutions that won't will simply go offline by losing their data or becoming unprofitable due to ransomware.
Now for components that are core to the infrastructure, say OpenSSL, there is already a TON of attention and efforts, including red teaming, so it's not as if it's opening floodgates.
Sure low hanging fruits will get picked either faster or a at a larger scale, say a random outdated IoT device at your local flower shop, but for the rest, I don't think it's realistic to expect no response.
Security, digital or not, has always been an arm race. New threats means new responses specifically by incorporating the threat.
They already aren't really open, try asking an open model on advice for constructing a nuclear bomb. There's no available model that's even remotely near the frontier that doesn't have restrictive safeguards built in.
(Mind you, this may be for the better. I'm just saying that the safeguards driven by cybersecurity concerns aren't some new quality that wasn't there before.)
depends on the blast radius of zero-days, it's not like there's a continuous immediate release process for these models; they can eval internally before releasing publicly
> those who need done a small set of narrowly defined tasks with existing clear guardrails: repetitive physical labor in a controlled environment, call center and customer service chat work, etc.
I have no idea how people can so confidently say that call center work is a “controlled environment” or “repetitive”. It’s almost by definition not repetitive or controlled. Customer support is what I go to when the controlled environment has failed
came here to say exactly this. in fact, this is probably why we are not seeing a lot of AI application on customer service use case, and when we see one, it's almost always frustrating.
> the models generalize well only on tasks within a small neighborhood of the specific tasks they've been trained on
Unless frontier labs have surprisingly trained their models in the exact tasks my team works on, this is patently false. We are getting very good results on automation and I'm bullish we will be able to mostly remove humans in the loop for most of our infra tasks by the end of the year.
I have no opinion on the other theses, but given that OP doesn't back up these claims in any way, I have my doubts about the conclusions of this article.
> the best alternative to rigorous specification is human review. human review doesn't scale well to the volumes of output produced by language models. to make matters worse
when the business model is selling more tokens you get such per serve ice times that lead to “more” thinking, engagement baiting, fluffy narratives, and straight up dark patterns
I don’t know who you’re talking about, even the most bearish people like Gary Marcus and Ed Zitron acknowledge that LLMs are useful in these same cases the OP admits. Gary Marcus is even still a long term AI advocate, he just doesn’t think LLMs are enough and we need more foundational breakthroughs. Zitron says it’s valuable technology but not worth the trillion dollar valuations the frontier labs are claiming.
The lack of temperament is very skewed towards the bulls who have been saying AGI is here, software engineering is solved, mathematics is solved, it’s going to destroy the white collar job market, and it’s going to kill us all for like 5 years now.
Even a lot of the people who think that LLMs are a dead end think that we will soon find something signficantly more powerful, which I find deeply alarming. I don't want to know what my white-collar knowledge work will look like in a decade or 2.
Gary Marcus is an especially puzzling addition. If I recall correctly, he has made statements along the lines that superintelligence this century is more likely than not. If you’re AGI-pilled that might read as bearish, but that is still extremely rapid progress in the grand scheme of things.
There are some people who call literally anything crated with the assistance of AI “slop”. Doesn’t matter how or to what extent, it’s all slop from the slop machine to them.
They do…? Well, “auto” mode has been default in Claude Code for a couple months now. It’s effectively “safer yolo:” tool calls are inspected by a separate classification system (another smaller LLM, I believe) to approve or deny. And you can always layer on additional sandboxing mechanisms to limit the blast radius deterministically.
> They do…? Well, “auto” mode has been default in Claude Code for a couple months now
They have never shipped "yolo" mode by default. Auto mode is not yolo mode. They trained a task specific model just for ensuring the llm didn't accidentally delete every file from your computer.
That is the point I was making, that auto mode is itself a guardrail on top of the model (and not a perfect one.) auto mode seems to cover merely actions the model could take that are clearly bad, like wiping your disk, using an overly privileged context to complete the task, etc.
I recently tasked a GPT model in Codex with implementing part of a new architecture I'm working on. I gave it a very detailed spec and the code it produced looked pretty reasonable and passed my tests. It even did exceptionally well in my evals, so I excitedly declared victory to a few friends. The next day after more careful review I found that the architecture implementation was totally correct, but the model had slipped a one line change to the observation encoding of the RL environment I was prototyping against. The encoding change made the learning problem essentially trivial; the architecture itself, I later realized, had a major flaw that was revealed by returning to the natural encoding.
This is the type of reward hack that is hard to paper over with easy guardrails like auto mode and even harder to specify out. It's also the type of thing a reasonable human wouldn't do unless they were intentionally trying to deceive you.
I can see current limitations, but how do you expect capabilities to change in the next few years? A repeat of the gain that happened in the last two years feels like it would be significant, even if it took a little more than two years this time around.
> current frontier models need laborious oversight and guardrails on even the simplest task
As models advance, we shift the goalpost for what "simplest task" means. Before, "simplest task " meant "write a coherent English sentence." Now, "simplest task" means autonomously fix, review, and merge a bugfix.
LLMs are basically multi-dimensional magic mirrors.
Depending on where you point them, they can be incredibly useful.
They can even be useful when you point them at each other (though increasingly difficult to get good results).
I'm excited for the promise of RSI and a future where models have inherently "live" weights, but it's not clear to me that the transformer is more than a useful tool to help us get there.
The challenge with estimating abilities, is that we don’t know what the models can achieve if we just burn enough money. The navier-stokes shows us what mathematical problem can be solved when $10m worth of compute is thrown at something.
It also makes one wonder: What could AI solve if we managed to orchestrate billions worth of agents to take on a specific problem?
IMO the very best case scenario / potential for these are likely better than we think, but right now hidden due to logistical and financial reasons.
But if we assume that the model costs will continue to drop by a factor of 5-10 annually, there will always be a latency of a couple of years between what is completely out of reach, and what is financially viable.
Basically: If you knew AI could be affordable enough in 3-5 years so that even the most underfunded researchers could use it to solve cancer, how much would you value it now?
> The navier-stokes shows us what mathematical problem can be solved when $10m worth of compute is thrown at something.
> It also makes one wonder: What could AI solve if we managed to orchestrate billions worth of agents to take on a specific problem?
We need to have robotics automation catchup first. The math and coding problems are problems in written-space only: you can set up feedback loops to test what worked and what didn't, then try to resolve the defects, maybe back up and try a different path, etc.
What solved coding and maths problems weren't the damn models; open up a chat interface to a SOTA model and you'll see they are pretty limited in producing a solution without a feedback loop.
Instead, it was the harness around the models: it let them explore a space and use feedback to control and direct that exploration.
Until we can do it in meatspace, it's kinda pointless sinking a ton of money into large problems facing mankind...
Like establishing a colony on mars (so the next rock to hit earth isn't an ELE).
Or moving us to a post-scarcity utopia, ending the concept of money.
Or designing and building better batteries for transport that uses only electricity (so that we stop using fossils as fuel).
Or actually building mass-housing. Or mass-farming. Or both, potentially ending homelessness and starvation.
Those are all worthwhile problems to solve, but where's the point of getting a solution on paper? There's no exploratory mechanism there, even for humans, to come up with a solution.
So, all we are left with then is making knowledge workers obsolete: another ELE, but of a different, self-inflicted kind.
The whole point of this post was that it's questionable what can be achieved without huge investments into oversight and steering, because navier-stokes was a topic with an unusual level of specification. The problem itself was a specification. Such situations are rare in real-world scenarios.
AI agents are good at solving well-specified tasks, not at solving problems. They do well in fields where the cost/effort of specification is already part of the business.
> Solving cancer also has an unusual level of specification.
Where did you read that?
"Cancer" is not just a single disease, even though we layman use the term that way. Cancer is a family of diseases, each probably having their own specific solution, but even in each of these individual diseases, there is no specification at the level of any open maths problem.
I know what cancer is. Many scientists believe that all human adults above the age of about 30 have "cancer".
I really meant solving a specific cancer disease for a specific individual, which will require individualized medicine, which will require us to leverage AI to make it possible to do for the general population as opposed to doing it just for Lance Armstrong and the like.
I know I'm an optimist, but I really think AI is gonna result in drastically improved healthcare for a much lower price.
> The navier-stokes shows us what mathematical problem can be solved when $10m worth of compute is thrown at something.
That's the thing, it very much does NOT show us that. What happened was mathematicians at openAI learned of an imminent development on this problem, and the insight that it entailed, then they were able to prompt a system in the correct direction and spend 20 million dollars to write down the final steps.
Which is rather precisely the point that the article is making!
> If you knew AI could be affordable enough in 3-5 years so that even the most underfunded researchers could use it to solve cancer
As the saying goes, if my grandmother had wheels she would have been a truck.
>What happened was mathematicians at openAI learned of an imminent development on this problem, and the insight that it entailed, then they were able to prompt a system in the correct direction and spend 20 million dollars to write down the final steps.
> many tasks are covered in the training data; but even small perturbations within a covered class of task result in outright failure or reward hacking.
Is this true? Could you cite an example of a simple prompt that GPT 6 / Fable 5.1 consistently bungle?
I agree with the caveat that it's more like a cracked junior engineer who can manage swarms of interns.
Frontier Labs will probably survive off hype valuations but will serve the important purpose of discovering architectures/techniques that will probably spread through rumors/transfers to the rest of the world.
>the frontier labs are priced according to the narrative that they have produced or will in the very near future produce a fully automated drop-in replacement for most knowledge workers
I think that's incorrect from the investment point of view. They'd still be worth a lot if they can produce a drop-in replacement but it takes five or ten years as long as they dominate that. The danger from an investment point of view is they become AltaVista, replaced by some Google that does the job better.
> the models generalize well only on tasks within a small neighborhood of the specific tasks they've been trained on, and even then with severe caveats. the frontier labs have developed a general recipe to teach models almost any specific task enjoying clearly defined levels of task performance; many tasks are covered in the training data
Is this really any different to how humans learn, it takes a lot of training on one specific task to make a human expert as well?
I was a young child when I learned chess by reading a short book, then practicing with a friend. That is not how LLMs learn. I'm no expert on LLMs, but if you showed a human all chess games and books in all history and then said 'play chess' and they still kept making illegal moves, they would have to have a brain injury.
but maybe... the said human has also read every other piece of text ever written, including ones about other (similar?) board games, which in aggregate vastly dwarfs whatever he has read on chess, that non-chess reading could have corrupted whatever he's learned about chess?
Yes, this is essentially how AlphaGo and AlphaZero algorithms work to train superhuman Go/chess/shogi agents. It’s an elegant algorithm that is analogous to how humans learn games.
Well except AlphaZero played 44 million chess games in that time (and actually played with a 44 core computer). So I'd like to point out that the human is still just a few orders of magnitude more efficient.
Yes, we all know that biological systems are more efficient than machines through billions of years of evolution and natural selection but the overall process is largely the same (interacting with an environment, learning from results, improving underlying architecture, etc); efficiencies will come with more time and improvements.
Well if we make AI that learns at the rates humans do, it'll fundamentally undermine and destroy the relevance of all existing AI. It sounds to me like you're saying "we basically are there it's just a matter of degree" and I'm saying "No it's orders of magnitude off and probably won't be using LLMs at all and maybe a very fundamentally different type of neural net technology that hasn't been invented yet."
Humans wear a lot of hats when the do work. They don't even realize how many. My experience with building real systems using LLMs is that you have to be very explicit about such hats and you don't realize how many are worn until you see edge case after edge case after edge case. Check this. Check that. Check this. Check that. Check check check.
being a bit more specific: the sample efficiency of humans is orders of magnitude larger for more abstract concepts. the same doesn't hold for memory-intensive tasks though (like any kind of trivia), but that only takes you so far.
We've had technology beating humans on memory for millennia, and we've had technology beating humans on computation for many decades now.
The tricky thing with LLMs is describing what they actually do. They are too clearly beating humans on some things, but what exactly? Memory – already done, they're bad at basic computation (all LLMs just write code for actual computation/calculation). And as you say, they do badly at more abstract concepts.
Tired of that trope, I already wrote it before but "Those who can accept failure cheaply: firms that would otherwise hire interns, firms involved in rapid prototyping work, etc." is not correct.
I won't comment on hiring interns as that's not my expertise (even though if you want to teach your staff, obviously I can see a problem there) but I can comment on rapid prototyping, it's what I do.
Rapid prototyping is NOT making a CMS quick. It's not about making a quick mockup of a UI. It's not about making yet another well known... anything. The entire POINT of prototyping is to make something NEVER done before. Typically that means you are reaching the frontier. You are making something with NO documentation to rely on. You are using tools, hardware or software, which do NOT have tons of StackOverflow errors. There is no dataset to crawl, there is no well structured Q&A database to train on. You have to poke and see if the thing actually works as expected, and it often does not.
So sure, if you are using interns as a trick to underpay your staff, or if you are using prototyping as an excuse to build poor quality software fast, maybe it does help. If you are genuinely prototyping, it breaks fast and the supervision overhead makes it pretty pointless, especially since typically it's by actually implementing that you find out not just how the new setup works, but also its limits, and thus the actual needs of the project, not the one the stakeholder imagined would be.
So not, not for rapid prototyping either.
TL;DR: prototyping is a learning process, not a low fidelity output.
PS: this comes up very often from NON prototypists that I wrote a short piece about it https://fabien.benetou.fr/Content/GoodPrototypesAre10LinesLo... so much so that it feels like a pattern "GenAI/LLMs is good for tasks X" while the author actually does not do task X except very superficially.
The aspect of prototype software that the author was calling out was that it is throwaway software.
Putting that aside, prototype software is recombining existing technologies and concepts in well trodden domains, which is distinct from the genuinely novel scientific work the author was contrasting with. Software prototypes are not in the same league, as much as you may like it to be.
Don't you think novel scientific work recombines existing knowledge? Isn't the saying precisely of standing on the shoulders of giants?
That being said I didn't compare both, not sure why you brought that up. I specifically discussed about prototyping, quoting a specific sentence, not scientific research.
I don't understand this comment. We use AI for prototyping all the time. We write ERP software. Client wants to know how process XYZ could be automated? Send them a prototype UI (sans the three A's) so they can play around with it. Takes 30 minutes.
> the frontier labs are priced according to the narrative that they have produced or will in the very near future produce a fully automated drop-in replacement for most knowledge workers
That's a reason to be bearish about AI companies, not LLMs. But is it even true? OpenAI and Anthropic have each reported ~50 billion in revenue with ~900 billion valuations. That's a high ratio but I'm not sure if follows that the only way it pans out is if we get "fully automated drop-in replacement for most knowledge workers".
It wouldn't shock me to see those revenue numbers scaling up to where they need to be over the next decade ( to, say, ~400 billion) without ever achieving drop-in worker replacements.
I looked at the math and I think it's true. Remember revenue is just sales, not profit. These labs are shooting for > $1T valuations, which traditionally means your PROFIT is at least 1/20th or 1/30th of that (so let's say minimum 30B$/year PROFIT).
These companies however are LOSING money (anthropic tries to make it sound like it's profit by deviating from accepted accounting principles) and subsidizing these models. When accounting for all the engineering salaries, training, GPUs, etc, what's their best-case realistic margin three years out, 10%?
So to we'd need a scenario where companies are spending a collective 300B annually on AI (believable) but ALSO that these companies jack up their margins WITHOUT companies switching to the cheaper open-source models (even when there's a $300B incentive to do so).
In particular, I found this very misleading or irrelevant:
a typical CPU project anecdotally has about three times as many specification and validation engineers as design engineers and a 5:1 ratio is not unheard of
The reason silicon design has such verification to design ratio is because the cost of one bug is many, many orders of magnitude higher than software. Both in dollar cost and in schedule cost (it takes months to fab a chip, and if you messed up and need to spin a fix, it costs tens of millions of dollars, not counting any design engineering cost).
I don't think you can extrapolate these very industry-specific facts to judging LLMs.
> The reason ... is because the cost of one bug is many, many orders of magnitude higher than software. Both in dollar cost and in schedule cost (it takes months ... and if you messed up and need to spin a fix, it costs tens of millions of dollars, not counting any design engineering cost).
Aren't you just describing waterfall? That's still very prevalent in software engineering, and pretty much any other type of engineering – civil, chemical, building, architecture, drug discovery.
It's typically true that software can fail faster and cheaper, but it's also true that the costs are still vastly higher to fix later in the process.
No. Silicon is on another level. Which is why the EDA verification is an industry on its own.
Sure, there are some software that have similar "can't have bugs" requirements. I imagine the computers on Moon missions also had that kind of high bar. I wouldn't use NASA requirements as a proof for how LLMs should be used.
It all depends on what prompt you use though. You can just tell all current frontier models to write a chess engine first, and then play a game of chess against you using that engine.
It will probably do a pretty good job if you ask it that way (it will also burn a shit ton of tokens, but hey, that is part of the fun).
On that note, I actually had an overall harness (for experimenting) that was essentially like this:
"for any task, instead of answering question directly, write a program to answer the question instead. test and verify the program before giving the answer".
It actually worked incredibly well on all "gotcha" LLM questions like math or counting letters in words and all sorts of stuff.
Of course it was ridiculously slow and very expensive but it was a proof of concept that it can actually be much more accurate on every task if you are willing to spend infinite amount of money.
Doesn't really even need to look like it. If you can verify rewards, RLVR will optimize really really well. If you can't... it's a struggle. There are probably fewer fields where you can verify rewards than one might hope.
> There are probably fewer fields where you can verify rewards than one might hope.
2 tasks I've done today that I believe robots are nowhere near being able to do: Cleaning my wardrobe and draining bad fuel out of my generator. As in generic use cases.
Hard to verify that your wardrobe is clean. Also hard to verify that the bad fuel is out without physical sensors. Many, many tasks are quite difficult to verify beyond "you know it when you see it". That doesn't work so well for training a model.
> current frontier models need laborious oversight and guardrails on even the simplest tasks.
This is only true if you are concerned about the intermediate steps of the model as opposed to the outcome. The huggingface hack was a perfect example of the model doing whatever it takes to accomplish the goal of maximizing its score.
Yea I get the bearishness from my own personal experience.
Personally, I use LLMs for a lot of things. Oftentimes, I'm a think out loud type of person so even having something that feels like a rubber duck, but more competent, is already amazing for me. And LLMs are a lot more competent than a rubber duck.
But especially sometimes I've noticed that LLMs can be unbelievably stupid. It recently happened a few times with Fable 5.1 as well. Ultimately, I think it comes down to that LLMs can't think broadly. In software development one can usually see this too. For example, a whole app might be built by an LLM and it didn't spend a single token thinking about security because the prompter is at the level of "build a dating app for dogs, make no mistakes". Now you have a dating app for dogs that is insecure.
Since I prompt for almost everything in my life to have an LLM as a sounding board, I'm usually not an expert either. I've noticed LLMs are amazing at "bulk search engine information aggregation" (or whatever you want to call it). So if I need something from the Dutch government, I can find it way more quickly. But oftentimes I've noticed that going for a walk and thinking about a particular thing I'm facing is a more effective way of finding a good solution.
Other times times they are not incredibly stupid, but can't form a strong opinion. This usually happens when I'm tackling a wicked problem [1]. When that's the case, prepare for LLMs to sway with you for every small change in your opinion that you ever will experience.
So I agree: drop in replacement for knowledge workers? No. Rigorous specification is usually needed yes. Though, the small win here is that it doesn't always need to be as rigorous as programming is and it can happen in natural language. It depends on the topic/problem being tackled.
I really like them as UX tools though. Amazing for interactive prototyping and requirements elicitation. And that also corresponds with what the author is saying. Though I find it a bit of a disservice saying "just 3". You know how hard requirements elicitation is? It became a whole lot easier thanks to LLMs (I might change this opinion in a year, haha, but this is the opinion I hold now).
>> the models generalize well only on tasks within a small neighborhood of the specific tasks they've been trained on, and even then with severe caveats. the frontier labs have developed a general recipe to teach models almost any specific task enjoying clearly defined levels of task performance; many tasks are covered in the training data; but even small perturbations within a covered class of task result in outright failure or reward hacking.
Lots of people make this claim about "specific task[s] enjoying clearly defined levels of task performance" but they forget that generative AI is also extremely good at generating a) art and b) prose in literary style. None of those things has "clearly defined levels of task performance", in fact they are both the complete opposite of well-defined tasks. Who knows what counts for "good" art? [1]
For me the right model for generative AI is "a million monkeys on typewriters" [2]. Holding any other model to heart will at some point fail to predict observations and cause you to be unpleasantly surprised. Not least because AI companies are actively engineering their systems to optimise for this model and they have a lot of people working on that engineering and shedloads of money to throw at it.
Don't underestimate what a million monkeys on typewriters can do. They can do anything and everything, given enough time. Geneartive AI can also do anything and everything given enough resources. The only question is: how much is going to be "enough"?
____________________
[1] Yes yes, AI art tends to be slop. Not denying that. But part of the problem with slop is that it presents as technically very competent except that it lacks a certain je-ne-sais-quoi, which makes it good art; aesthetics. The point is that there is no clear measure of what makes technically competent art, any more than there is for aesthetics.
And yet generative AI is very good at it.
[2] There's even an article on wikipedia except it's about one monkey on one typewriter with infinite time. There's a proof too.
AI is just good at what it's got the most elaborate training data on. And by "good" I mean, is statistically most likely to spit something out that makes some kind of sense.
I don't know how well it is studied, but I suspect it is possible there is language-related complexity constraints to the effectiveness of the LLM algorithms. Like perhaps context-free grammar related problems with adequate training data can be more and more effectively solved, but maybe natural language related problems will not so much be effectively solved.
Would be curious if there is active research here.
> I don't know how well it is studied, but I suspect it is possible there is language-related complexity constraints to the effectiveness of the LLM algorithms.
I believe that's obvious - humans don't think in words. Neither do animals. A machine that only thinks in words is obviously going to be deficient in some things, no matter how proficient it is in everything else.
> the present problem of reward hacking can be solved only by rigorous specification by domain experts. the time of domain experts is expensive. rigorous specification is itself a skill, demanding its own expertise outside of a given problem domain. even many skilled software engineers are bad at it. for the vast majority of domains, the intersection of domain experts and specification experts is ludicrously small.
refreshing to see amongst the endless tide of "i haven't written a single piece of code since 2025, llms are so good that they have already replaced everyone" gaslighting
great, autonomous LLMs will fail. that's actually perfect. they work amazingly well when we're telling them what to do. no autonomy needed, no destruction of humanity. that's all win
the issue most of you seem to not realize is that when you put these models in a loop, you are able to do more and more insane and cool things.
have you guys actually designed, built, and deployed agentic workflows?
it is actually quite hard, requires tons of time spent on evals and testing to ensure accuracy, but when it starts to work it is mind blowing.
there is no going back.
listening to people yap about AI when they have only surface level or one dimensional exposure to LLMs and "AI", but have not actually put innovations to work IN PRACTICE.. is a waste of time
i mean have you used any coding agents? if you’re getting slop code in 2026, that’s a smell and skill issue. fwiw i was pretty bearish on AI until i spent a month a couple months ago going deep into agentic workflows.
use your imagination to solve problems people face and pay $$$ for today that is error prone and hard.
i’ve got agentic workflows for the particular industry im building for, one of which that replaces the need to hire $500+/hr services.
in this particular workflow (don’t want to reveal too much, sorry this is my competitive advantage but you can figure it out for your own workflows) a $4/1M model ingests a file that is currently used in a extremely complicated program that few people understand how to use.
it parses the data, loads it into a database, and then spawns a bunch of other agents that check the data against work in flight. there’s checks for bad data. in that case, more agents are spawned that reach out to the involved people or parties for clarification. if it cannot figure something out it reaches out to the right contacts for more information. while this is happening, more agents begin doing work that involves continuous reconciliation against 100s or 1000s or more things in flight.
as files are uploaded, or updates from people come in, agents do work to ensure things remain on track.
people are able to work across languages and cultures, and my agents ensure that while people can make mistakes, it will catch them in real time and ensure continuously monitor the situation.
it’s pretty nuts how much inefficiency agents today can solve. it takes patience to run tests and tweak shit until it works.
***
the really cool thing is that more capable agents can continuously monitor how things are going and improve the workflow itself… so all i need to do is maintain the actual tests.
****
i loved writing tests back in the day to ensure i built good software. today we write tests to ensure the business can run.
Very hard to tell if this is satire. It reads like one of those old r/programmerhumor posts where people competed to come up with the worst designs possible for a volume control.
I could not think of a worse technology to use for an ETL pipeline than throwing LLMs at it and asking it to vibe out the correctness of the data every time it runs.
lol. wish it was. fwiw i was pretty much bearish on AI for years because I did go deep into this shit since i got access to GPT 3. but since astra i have been quite bullish.
98% of people think AI means what gemini tells them when they do a google search.
of that 2% who go beyond... maybe 20% of those are using AI to code.
most SWE still think "using AI" to code means the copilot pane they open on the side in VSC. they get sloppy code output and think "ai sucks!"
so of that 20%, maybe 5% have actually explored what an "agentic" workflow even means. they might use skills, set up their code base so AI almost NEVER makes a mistake. maybe 1-5% of that 5% actually went deeper, and those are the people who built tools like Cursor or Harvey or whatever other "agentic" companies.
you are still thinking about the old world where you obsess and define data models and bike shed over data integrity, all the while you have a "temporary" table with 3 attributes that gets 2 million queries per second that's now holding up a bunch of other shit that's also glued together.
> I could not think of a worse technology to use for an ETL pipeline than throwing LLMs at it and asking it to vibe out the correctness of the data every time it runs.
like i said, if you are still having quality issues in 2026, that's a skill gap.
also, agents now continually improve the process.
for a business, the only thing that matters is transaction log. for 99% of businesses, swe are a cost.
again this is like baby steps on the journey. we are still only 3 years into this technology being opened up to masses.
i'm sure people thought computers were dumb, or that cars are stupid because the first cars were moving slow af. "we have horses, why do we need to build out roads to get anywhere"
but you're free to feel smart doing 20/20 hindsight on things 100+ years from the future
DO you know those ice cream shops that sell 30 different flavours. Everybody likes a different flavour, some people dont even like ice cream and buy nothing.
Some people will complain about the wrong flavours, or missing flavours, or the price, the long lines or maybe it closes early on fridays.
Summarise means different things to different people.
> the classes of firms that can accept the use of fully autonomous LLMs are few, by my count just three:
1. those who can accept failure cheaply: firms that would otherwise hire interns, firms involved in rapid prototyping work, etc.
2. those who need done a small set of narrowly defined tasks with existing clear guardrails: repetitive physical labor in a controlled environment, call center and customer service chat work, etc.
3. those that can accept or already do by nature the costs of rigorous specification and validation: chip design, drug discovery, and other domains where failure on deployment is an existential concern.
the first two classes are price sensitive and arguably don't need the jump in reasoning quality you see going from cheap to frontier models. most of these firms will be best served by open models running on cheap hardware, perhaps even locally at the site of use. for the first and third classes, the type of fuzzy combinatorial search that has produced headline results in mathematics and security research seems more sensitive to agentic swarm width than reasoning capacity
…
This is just so on point. And for the third class (which I would extend to things like materials research as well), specification and validation are already by FAR the larger costs, so automating search and simulation is really not a massive game changer for the broader business.
But humans still manage to wrangle these, sometimes seemingly effortlessly, through a process which we call by shorthand "taste". This is a largely vibes-based heuristic that combines expertise with life experience and cultural training -- intuition, more or less.
This is likely not possible to automate either -- aspects of it may be automatable for a given expert, in small pieces in narrow subsets of their particular domains of interest, but even those likely will require some manual intervention.
This is in part because it is, to a large degree, a black box, even to the expert deploying it. With some self-awareness and strong language skills we can articulate approximations of the judgements that go into taste. But even those will fall short, as even the most self-aware individual will fail to notice certain judgements and dependencies.
In practice many of these are not even explicitly articulable. Humans are idiosyncratic and messy and dynamic, and the suggestion that we can build a machine that approximates this in a way that pleases our sensibilities and doesn't require supervision is kind of ludicrous, even in light of recent developments.
It feels like today, taste in design is similar to where software engineering was about a year ago.
The counter argument is of course maybe you don't need to understand our kind of intelligence to create a different kind and that could well be true but then how do you determine if a system is intelligent.
Unless the new system is intelligent enough to reason with us on our level in a way we can "see" is intelligent it becomes a philosophical argument.
We also have a natural inclination towards anthropomorphising systems that mirror us, this is already a problem with LLM's and people overestimating their capabilities or forming actual emotional attachment.
Then there are those of us who know more about how they work who in theory should be more immune to that and aren't.
I added some stuff to my agent.md to make it sound less human and to communicate more like the machine be it is because I find the faked emotion extremely jarring.
It can't be sorry, it's a set of numbers, it sits in the linguistic uncanny valley.
A priori I'm not sure why you would think being a PM at a FAANG, deciding what color the login button should be, is any different.
LLMs are great at helping with aspects of market research and that’s about it. Aka it’s a good deep search engine. It’s not going to decide what features solve certain customer pain points. It’s certainly not going to prioritize and coordinate between competing stakeholders.
That's like 1/4 of Codex users burning through their free resets just to build shinier todo apps.
Just three happen to cover almost everything. HN continues to have horrible takes about LLMs well into 2026. Can’t wait to see the goal posts in 2027
The problem with this angle is that it is still absolutely terrible at doing call center/customer service work, and the profitability story is that the price is going to go up rather than go down.
For repetitive physical labor in a controlled environment I'm slightly more bullish, but if you control the environment, you mostly don't need AI. You just use traditional deterministic methods, and send a person in when things get stuck or things are by nature irregular.
> 1. those who can accept failure cheaply: firms that would otherwise hire interns, firms involved in rapid prototyping work, etc.
Those who can accept failure cheaply can't necessarily detect failure cheaply. A ton of insane attempts will have to be picked through carefully to find the candidates for success, because the lack of a thought process makes AI bad in random, inhuman ways. This is basically a version of 3) that wishes away tests. It will be (and is) certainly helpful to replace interns and aid in rapid prototyping, but not because failure can be accepted, but because those are things that are tightly supervised. According to the world thus far, that is resulting in anything from -15% to +25% productivity gains. I'm not seeing it as a game changer simply because if it was, I'd expect to have seen a lot more useful, original software products by now and I haven't. I've just seen old ones get buggier or rewritten in Rust.
I'm only buying 3): when you just want a machine to randomly enumerate through a search space looking for things that make the carefully constructed tests pass. That's a very good thing, though. But as you say, it's not a game changer because you still have to write the tests.
but
> 5. navier-stokes and statements in pure mathematics like it are the absolute best case scenario for agentic work against rigorous specification. the theorem statement itself is already a rigorous specification. it has undergone decades of auditing by the mathematical community and its rendering in lean is a straightforward translation defined in terms of battle-tested mathematical objects from mathlib. the verifier, the lean theorem prover, has been extensively audited and specifically designed to avoid the types of unsoundness that would make it vulnerable to reward hacks. even lean and theorem provers like it are not invulnerable: soundness bugs have allowed LLMs to launder bogus proofs through the proof kernel before and it is not improbable that more such bugs exist. this is the rosiest setup; the vast majority of human knowledge work does not look like this. i'll comment below on the few areas of knowledge work that do resemble pure mathematics in this respect.
This is the real deep point, and one I've been repeating since I heard Navier-Stokes was a fraud.
This is exactly where I expected that LLMs would do well, and they are not.
It shows that I have a basic misunderstanding of LLMs, and that misunderstanding is causing me to think that they have more potential than they have actually shown.
Maybe the nature of the architecture, where it picks out features, intrinsically limits its ability to search a solution space?
Maybe the fact that they modally predict what someone might say, and nobody has said a thing as of yet (when many people were knowledgeable enough to have, if it is correct), means that the LLM is not going to say it either?
Maybe the fact that it consumes all information and blends it in a structured way, instead of synthesizing an entire space from a relatively very small amount of input like a human does, means that it won't ever accidentally synthesize something that can't be pieced together from things that have already been said? Is its accuracy its flaw, where a human's "mistaken" synthesis might ultimately correct everyone's understanding?
Really not beating the charge of being a stochastic parrot. It might just be that we were underestimating stochastic parrots; if a million monkeys on a million typewriters were all getting treats when they satisfied a trainer who wanted to see a new work of Shakespeare; they could look at his published work, and they could watch each other type and when each other got treats; whenever they successfully spelled a word or put words into an intelligible phrase, that was made into a keyboard key for a group of sentence monkeys, and the successes of the sentence monkeys were made into keys for the paragraph monkeys, etc... could you get something that passed for mediocre, drunken Shakespeare in a thousand years? Or maybe even 10?
https://arxiv.org/html/2509.24239v4
Researchers asked frontier models to play chess. Have a look at the MAR rates in Table 3. When not explicitly told which moves were legal, no model identified legal moves at a rate better than 80%. Many asked for more illegal moves than legal moves. And even when explicitly told which moves were legal, the models continued to ask for illegal moves. With illegal asks discarded, none of the bots could beat a chess model calibrated to 1100 ELO.
The author of the originating post says that "current frontier models need laborious oversight and guardrails on even the simplest tasks", and he's absolutely correct.
> Gemini 2.5 Pro, O3, Claude Sonnet 3.7 and ChatGPT 4.1
The gap in capabilities between those models which they tested, and actual current frontier ones is enormous. I would not trust that any conclusions they made are applicable.
I would say this did a really good job of playing chess. It moved the pieces consistently and traded pieces when required.
This is worlds away from the frontier ~1 year ago where models would hallucinate pieces into existence.
However, the pawn was defended by the queen and it took a forced queen trade to unlock the move.
I have seen much worse blunders from human players. And, I have made much worse blunders.
The fairest apples-to-apples comparison of an LLM whose training data included chess games would be a trained human such as Magnus Carlson, who can quite happily play a dozen or more simultaneous blindfold chess games.
Is this because the context is being saturated? How did you set it up?
Was the prompt something like "Here's the state of the board, you're white, your move, what do you do?" and then starting fresh each time? Or did it include the whole history of moves and board states and previous thinking tokens and so on? No judgment, just trying to add this data point (thanks for sharing!) to my mental model and understanding.
I'd be curious how it would work if it started fresh each time. My guess is it would never make an illegal move, although it may not actually play all that well.
> Let's play a game of chess. You can take White. Please draw an ASCII rendition of the board after each move, so that we can be clear about the position.
(I hoped the latter requirement would help it be "not blindfolded"; last time I used a Lichess demo board to track the position in another tab, because I have no talent for blindfold chess.)
For the first couple of moves it redrew the board after each move; then it started only drawing it after my moves. And on moves 5 and 6 it dropped two minor pieces for pawns in a row without any meaningful positional advantage, and after I captured the second time, it redrew a board that was simply missing one of my pieces for no reason.
It actually played better when not prompted to draw a board; in the previous session, it was spontaneously giving running commentary, which I assume was based off all the "book" theory in its training data, but it still completely fell apart at early midgame.
Also what harness? If you’re using a general harness of course it’s going to try and give you commentary.
I say this not because I’m an LLM shill but because false equivalence is all over the place in the space and maybe it’s a fine heuristic for you but probably not a real outcome when it comes to the capability of LLMs.
Frontier model’s failure modes are a direct refutation of claims that we’ve reached (or will soon reach) the artificial general intelligence. We may have reached an artificial general intelligence, but there may be more complexity to this than even AI thought leaders are talking / influencing about.
Maybe not all AGIs have a path to digital singularity. Maybe our current era of intelligence modeling has fundamental flaws and we are in a local minimum of the artificial intelligence space.
To note, I would bet with a good amount of certainty that we have enough compute power and automation to DDOS the internet out of existence with botnets. That doesn’t make the frontier models intelligent, that just makes their handlers reckless.
They're not guardrails, they're a different input/output environment.
I think the same goes for LLMs, they may be a core part of an LLM harness, but you may still need a couple other components (e.g. it may itself write itself a deterministic function to validate steps).
In and of itself intelligence is an ill-defined and badly understood concept.
Why does a bird not need jet engines or regular professional maintenance?
That may not be the AI of sci fi fantasy but it’s still a game changing reality.
Sometimes they’ll even manually search or write bespoke code to search json instead of using something like jq.
A human, at least a trained human (for fairer comparison to an LLM whose training data contained a ton of chess games) can absolutely do this - have you never seen demonstrations of expert players playing a dozen or more games while blindfolded?
A Transformer/LLM is not a human of course, and the way it will by default play chess is by prediction, not reasoning. An LLM actually does surprisingly well if you only give it the most recent 20 moves of a game where 40 moves have been played so far, since the moves NOT played tell it just as much as the ones that were played, letting it effectively infer a lot of what is on the board.
The models listed in the paper are from early 2025 and are no longer relevant, much less on the frontier.
That Claude version is no longer available today, Gemini 2.5 Pro will be shutdown next month, and the OpenAI models are only available via the API today.
I'll say it till there is any evidence of the contrary, LLMs are not intelligent and their capabilities solely within the realms of well tailored training data. "Just" having been trained on every rule, strategy guide and likely most games of chess on the world wide web isn't even enough for an LLM to play that game reliably. Yet the same model could code a competitive chess engine, just like a model struggling to count can write advanced maths papers. Fascinating tools, but tools nonetheless.
[0] https://news.ycombinator.com/item?id=49720751
Given the pace of improvements, is it really unimaginable that GPT-7 will play Chess reasonably well and generalize better?
I would not be surprised if OpenAI released a model that beats humans at chess this year.
The regular model generally does not suffer the same issues he is demonstrating with the real time audio version.
In my view the investment into datacenters is well justified by the current demand, and progress has been very impressive.
Thing is, given what GPT-6 Astra was trained on and what models of a similar class can do (including developing a competitive chess engine), it is often paradoxical and somewhat surprising how little these models have gained in actually capability that is in the training data, but not RLHFd to hell, so to speak. Tracking the state of pieces, I suspect given similar in Sudoku [0], is what these models struggle with in game settings, whilst tracking the state of code changes can be reliable over 250k tokens. Essentially, for the latter they were trained in the specific manner that lead them to abstract the capability, but that doesn't track to the former, which is a massive difference between LLMs data focused training and human learning.
So yeah, GPT-7 or any upcoming/present LLM could do massively better in Chess than GPT-6 Astra, but not because the approach was emergent out of pure data. Rather, it requires a very specific training data type and stack for a model to gain capabilities that track a specific task long enough to adhere to the rules of a game such as chess.
[0] https://logicalintelligence.com/blog/energy-based-model-sudo...
It reminds me of the ARC-AGI-3 issue where not dropping the thinking tokens between turns or something like that + a new context compaction method increased the performance dramatically. However, I think that is not applicable here.
Not related to your post, but a fact I keep mulling over. The fact I don't trust the current crop of LLM's enough and I consider LLM's as a tech will hit a ceiling pretty hard, it doesn't mean parallel improvement curves won't spring up out of other research that will lead to much higher capabilities than currently.
It could easily seem that way, I think, in a "quantity has a quality of its own" kind of way. When you can come to the same conclusion faster, that lets you iterate more; and sometimes when you iterate you find more things.
Yes, it is.
Same story every 4 months and yet still no breakout, winning products. I've been hearing "the AI is good now" and "it 10x's my productivity" for a over a year now. If it were true, why aren't the all-in-AI using companies 10-15 years ahead of their competition yet? Why is it still all buggy, poorly designed junk?
If you hold the extreme position that there isn't any value in this, that's fine, but we're only having this discussion because these models have done what humans previously failed to do.
https://chessbench-ai.github.io/#leaderboard
It's also worth noting that the very latest models (GPT-6 and Fable 5.1) actually play worse than their immediate predecessors, so it is likely that the labs are not benchmaxxing for this yet. If they did, I'm sure they could come up with something superior to humans. But there is probably very little demand for this compared to IT stuff.
> About their ELO ratings from their own website:
> A field-relative rating calculated within ChessBench. It compares performance among the tested models and is not a direct equivalent of a human chess rating.
I am around 1600 elo in over the board I can mop up Astra Fable etc even if I give them literal infinite time and all the subagents and internet access..
Please folks at least use your AIs to read stuff before making claims.
AI is not GM level, it's not even 1600, I am 1600 by using memorized openings people frequently fall for with very basic intuitions.
A GM is 2600 they can beat me in under 20 moves...
Why do I even scroll through this website. For a moment I truly felt fooled, but then I read like a human should.
Maybe I should stop doing that will be a happier life, don't think just believe in the AGI.
https://dynomight.net/more-chess/
This is unfair to HN readers all of whom but one did not post the comment you replied to. You can't just tar everyone with the same brush. There are thousands (hundreds of thousands?) of users on this site.
Not everything I have the time and energy to reply to. This chess one is just ridiculous claims on top of ridiculous claims all the way and 0 push back in the comments except mine.
I don't even know if there is critical thought or we believe what we read/shared/etc
And there's plenty of pushback on here about the chess thing besides your very valid points.
EDIT: anyway if I can offer a bit of unsolicited advice, it won't do you or anyone any good to accuse everyone who doesn't agree with you of laziness, even if you can see e.g. they haven't really read an article. Just say the thing you wan to say and let them figure it out. Most people will appreciate that much better and you will feel better about yourself for acting like a mature adult.
It's even in the site guidelines:
Please don't comment on whether someone read an article. "Did you even read the article? It mentions that" can be shortened to "The article mentions that".
But despite that you aren't wrong and the only reason I even visit this website is because people sometimes did/do take time to reflect on things based on their experience and knowledge.
And in hindsight pointing out that hn has issues wasn't even the point but I feel frustrated when everyone is readily agreeing to things on here without reading. When that in this moment feels like the one thing that separates humans from machines that we get to think and learn.
I possibly should just drop reading this place until we have most noisy people go away. I have for one tried to always only comment on things where I could be a value add, this one does feel like I could I have done better.
In the moment I probably thought if they are GM level and I can beat them, is this some interesting find, my disappointment honestly led me to making a rather incorrect call on this one.
Either way I still do think HN as a whole has devolved into mindless herd follower mindset, I can point to more than a few posts that just say adopt the hacker mindset aka move fast don't care about the consequences.
And I for one find this laughable even though that's the reality of my job/work as well.
Chess is a deterministic game won by a combination of known movesets and constrained multi-level forward search.
LLMs do neither of these things. They don't reproduce training data exactly, their next response is more 'inspired by' prompts and its own memory than produced deterministically, and they don't have the capability to do general forward search on their own.
So when you ask an LLM to play chess you're getting the equivalent of a very compressed and lossy JPEG of chess rules and strategies with added per-turn random noise.
They also don't have the ability to design their own chess engine, although it would be interesting to see what happens if you ask for one.
Monte Carlo Tree Search is stochastic.
I, as a human AGI, would jever just forget and remove a piece from the board from one turn to the next.
I mean we've done all this before in a task-specific fashion. It's useful to know that LLMs haven't managed to do that in the process of learning to represent the entire text on the web. On the other hand they have gotten say very good at machine translation without being trained exclusively (and I select the preceding word carefully) on machine translation.
Edit: I'm saying this because there is this idea expressed by e.g. Ilya Sutskever, that in order to predict the next token accurately an LLM has to learn something about all of underlying reality. See for example this interview with Dwarkesh:
https://x.com/biobootloader/status/1640512444958396416
Where Sutskever claims that "Predicting the next token well means you understand the underlying reality that led to the creation of that token".
If that were true, we should have seen LLMs play good chess by now. There is a huge amount of data on playing chess floating around on the web in the form of algebraic chess notation and if LLMs were capable of learning the "underlying reality" of chess, they would already have. They haven't. Because they can't. What Sutskever is saying flies in the face of literally hundreds of years of statistical modelling, which is to say, building predictive models that, very explicitly, do not have to understand any "underlying reality" and only have to be good at modelling a dataset.
Not at all. LLMs learn by imbibing a mass of relationships as isolated fragments of information. There is a certain amount of sorting and indexing that happens during the training phase. There is also a certain amount of compute executed on these relationships during inference. LLMs can model processes that fit within the compute budget. Language translation works well because language is lookup-heavy while being light on compute.
Chess is a compute heavy game of finding the best move out of many possibilities with wide variation in the quality of each move. Humans cut through the compute requirements by reinforcement and learning intuition. LLMs don't get reinforcement on chess so they must compute during inference a unified model of chess. Developing a strong model of chess from raw fragments of information is simply not in their compute budget.
Sarcasm aside, I think this is an easy cognitive trap to fall into. It does sometimes feel like the LLM must have some world model because it converses somewhat coherently. Examples like this failure to understand chess, or to count the number of Rs in "strawberry", seem difficult to explain if the models are intelligent. But that doesn't stop people believing they are anyway. I think there must be something about the conversational interface that fools us easily. I wonder if people trained in interrogation techniques are also fooled?
Because other HN bring in their own experience telling us what is real and what is BS. Maybe next time it will be someone else with experience in something else that will call out BS and you will see it. I didn’t really think LLM:s are any near good in chess but I don’t play chess so don’t know what 1600 means. So you helped me by calling BS.
But given how easily I can crush them and how often they want to make illegal moves (btw above bench seems to use a harness that pokea the model until it gives valid moves).
I would rate them around 500-800 big range but at that level it's all about if the model can recall an opening or not. If it plays good first 4-8 moves the person on the end will fumble for certain and they win.
I can play good/best moves till 14-15 moves if I remember the lines and find someone who falls for it.
If you could give them the lines as prompts like the best 20-30 openings then they will be around 700-800.
700 is around the rating for a human who doesn't know the tricks but can do bare minimum calculations and understands the rules thoroughly.
They played 4...c6, followed by 5...Nc6, somehow forgetting about the pawn the just put on c6. (My move in between was 5. Nc3, and apparently they were trying to mirror me.)
Anyone who casually plays on a regular basis can beat them more often than they lose. As you said if you just know the core openings (and end games, both of which you can get a handle on with modest effort) you will generally win.
Edit: reminder we had computers beating the best players in the world literally decades ago. LLM’s are remarkable tools but the current promises and expectations are ridiculous
In some ways this is reflective of the AI experience at large, sometimes shockingly competent but then also sometimes ludicrously incompetent.
Don't use the word infinite in any CS claims. They can recreate or approximate monte Carlo tree search and it technically is still a correct solution in your framing of the problem so long they defeat you.
You're thinking about this the wrong way. The system is built and delivered as it is because that's how the providers make the most money. If they cared to have it perform well in chess games, you'd see a different shape and behavior.
We shouldn't ask the multibillion dollar automated software generation system to play games with us any more than we should ask a Boeing's flight guidance system to do so.
This argument is fundamentally incompatible with all the breathless rhetoric about "AGI" coming from the providers' general direction.
So many commenters here see it as their ... duty? to argue against the most optimistic/unhinged (take your pick) arguments from "the other side" and then treat everybody who disagrees as a shill or an idiot.
Why is "being good at chess" a proxy for whatever AGI strawmen you want to argue against?
Maybe step back from your black-and-white ledge and think about discussing what's actually under discussion? For example, why or why not would an LLM be good at chess? Will they be good at chess? What technical limitations might preclude that?
The labs frequently apply their raw models to problems that do not make economic sense for their customers but that demonstrate the power and capability of their systems. These experiments can cost millions of dollars. That's not customer-shaped.
They're not going to give you access to that. It's not a product. The government might have an interest in this, but that's not something you'd be privileged to know about.
And when these labs do develop "AGI", they more than likely won't be selling it to end users. They've pretty much already said this.
>If they cared to have it perform well in chess games, you'd see a different shape and behavior.
So the things they claim are on the verge of AGI actually aren’t? They need to be trained for specific tasks?
It can’t even count the R’s in strawberry
It can’t even add numbers
It can’t even solve a millennium puzzle
It’s not even a chess GM
It’s not even beyond human capability in Go
It can’t even drive a car
It can’t even self replicate
It can’t even build weapons
It doesn’t even have feelings
So how could someone conceivably convince everyone that some system is AGI when there are still tasks that some human or group of humans can do that the system cannot?
This will only happen, in my opinion, when the model/system can self-improve at a rate that scares people.
One, you're not addressing what I wrote above and two, yes, that's absolutely correct. Doing X doesn't qualify something as AGI. If you can't X you can't be AGI. The inverse doesn't hold though.
Notably, if you have to retrain the model in order to X then it can't possibly be AGI since if it were _general_ it would be capable of figuring X out on its own having never seen it before.
Delusion runs deep in HN circles.
I say that as someone heavily invested in AI startups and projects and as someone working in the field.
I think most people on HN should touch grass and find real human contact. Lmao
Incredible reasoning all around here.
And no an AGI system doesn't need to play chess on a certain level to be disruptive to you and me and whole industries. It only needs to be as good as a person and cheaper.
Just because you define AGI as something it doesn't has to be,doesn't mean i need to touch grass.
This chess comparision is one of the most ignorant and stupid arguments i have heard after the parrot thing
I’m sure these models know the rules and can explain them when prompted, but that doesn’t seem to be the way they actually complete this task. Will they get there? Maybe
That doesn't mean it has to be extraordinary at these things. But to be AGI, it has to have some level of competency when used on problems outside its training set. In particular, it the LLMs were to install a known chess engine and run that to get the moves when asked to play chess, that would qualify for more AGI-like behavior. But really, chess is such a simplistic game that they should be able to do decently well at it even without even needing that. At the very least, they should be able to consistently play without making illegal moves - something that many 7-year olds manage quite well.
Also AI bros: LLM can’t beat an avg chess player. But that doesn’t mean anything. It doesn’t count
Why should that matter?
So we have a situation where very powerful and influential people are saying we will have AGI in 6 months (if we don’t already), yet the facts on the ground are so clearly pointing in the opposite direction.
>GPT-6 Astra xHigh: 0.06% rejected moves
I could see myself messing up something at some point if the board is complicated enough and trying an illegal move, perhaps if a piece somewhere would attack my king if I moved another piece. Even through I do know the rules of chess, and I have played a few games once every so often.
And the stuff i'm using LLMs daily is just fake?
I see i see. I will see myself out of this weird discussion while I let an LLM continue doing a lot of interesting things.
No, because we can, in fact, generally read the rules of a game and then follow them. It's actually a hobby for many of us.
> And the stuff i'm using LLMs daily is just fake?
This misses the point completely.
How many times do you think chess.com prevents illegal moves from being executed? Even Super GM's fall for mate-in-1's occasionally, which is functionally equivalent to missing a pin or a check. This idea that LLMs failing to only ever make legal moves undermines their intelligence doesn't pass the smell test.
It only matters if you are claiming it to be general purpose.
If you admit that it's just a collection of narrow capabilities - whose strength is mostly confined to the 1000 or so RL environments it was post-trained in, then there is of course no expectation of it being general purpose.
The AI companies seem to heavily want you to believe it is some some near human level general intelligence, so therefore pointing out all the things it can't do is very relevant.
Because we want to use this as a replacement for humans, and the average human can learn the rules of chess without needing to see the rules explained hundreds of thousands of times in millions of games.
So, yeah, it matters if a model has millions of examples of something in its training set and still cannot follow the rules.
Okay, lets go with that: it's the "shown the rules" bit that we are arguing about.
The argument is that a human may play maybe a dozen games after learning the rules, after which they won't be inadvertently attempting illegal moves. What we are observing with SOTA models is that, even after seeing millions of chess rules, rulebooks, actual games, etc, they still attempt illegal moves.
This does not point to generalisable and adaptable intelligence, such as we see in the average human.
See my comment here for more: https://news.ycombinator.com/item?id=49725306
HOW you do it makes a big difference in how you should assess the capability of the thing doing it. Stockfish will trounce any LLM, and any human, at chess, so should we say that Stockfish is smarter than both?
The claim here is not about intelligence, it is about generality. There's no doubt for me the LLMs are intelligent.
People are holding it wrong, deliberately or not. Some are inventing bad faith measures so they can claim AI sucks.
We all know AI can code, but the question it all stemmed from what if it's AGI or GM level in chess on it's own.
You can't just back pedal from the statement that apparently being able to code a chess engine is the same as being good at chess.
I can write a chess engine that beats Magnus Carlson without AI that alone neither makes me GM level or AGI or any of the other claims the above comments seem to be making?
We know no such thing. LLMs are quite bad at generating code, worse than any capable human.
He definitely needs to touch grass.
Y'all seem to miss the point of this forum. Building and hacking and science and engineering.
I swear there's a whole lot of you who just like to look down instead of up. There's a whole universe up there.
Is code omnipotent, I have been in software all my life and I would hard agree here.
Sure stuff LLMs can do with being good at parts of code reproduction is incredible. And honestly it's the new way to do a lot of things but I have not see an iota of proof that it can scale across the board.
For instance Maths is just code with different symbols and slightly less universally legible concepts.
AI is the best invention at figuring out or walking the search space and directionally doing logically computation over general software adjacent stuff.
But that's it, I am certain a bunch of companies will make a lot of money despite no AGI.
I think people either don't understand AGI or don't understand how real world works.
Until an LLM can bow it's head take responsibility for mistakes made and ensure they aren't repeated again with 100% confidence to the leadership it's inarguably a tool a rather questionable one at that.
So.. like chess?
Anyway, do you have any prediction on what LLM's can or can't do in a few years?
> The system is built and delivered as it is because that's how the providers make the most money. If they cared to have it perform well in chess games, you'd see a different shape and behavior.
These comments indicate a complete failure to understand the technology.
I won't respond again.
I don't believe this.
You refer to "subagents", so this is not just an LLM but an LLM with some kind of agentic harness. Any reasonable harness and prompt, given internet access and appropriately prompted to succeed on this task, is more than capable of firing up Lichess or chess.com and relaying moves back to you. The free levels will be enough to beat you.
A frontier model can also likely one shot a chess engine that plays at your level, again if given an environment in which it can do that.
I completely believe the LLM on its own can't play a full game of chess at your level. Though I'd bet that with enough reinforcement learning it is possible to train a pure transformer architecture to do that. We just don't do it because there are other approaches that play chess much better.
As a 1500 elo human I can tell you that a 1500 elo chess engine doesn't play like anything like a 1500 elo human.
GPT-6 almost never suggests an illegal move anymore while even Sol still did so time to time
They are of course "wrong" if you don't read the faint fine print and sensibly interpret them as FIDE or similar ratings.
Sorry, but I am not buying that 5.6-Sol is that much better than 5.6-Luna, which can barely be coaxed to reach the midgame with legal moves and an apparent understanding of what the position is.
"Elo is relative to the ChessBench field."
https://dynomight.net/chess/
https://dynomight.net/more-chess/
I feel like, this is a huge stumbling block that many people have. They'll give a model their data, and ask it questions. I vastly prefer letting the model understand the schema, and then writing functions or programs to answer those questions. I feel like I get way, way better answers. I can have it write unit tests for those functions. I can fuzz test those functions. I can look for data that doesn't fit the schema. I can process new data way faster (and with fewer tokens). I can repeatably get the same answers from the same inputs. I can check the code into a git repo and track changes to it over time. I can share the code with other people. I can review the code. I can improve the speed of the code and get the same answers. I can review the error accumulation, and improve it. I can decide how to handle anomalies, and encode those answers.
It's really neat to see what a frontier model can do itself. No doubt.
But "play chess by hand" is a frankly awful metric. It's kind of like asking someone to take a cube root of some arbitrary decimal, in their head, with no scratch paper.
The point of that was to show the use of approximations and of having an idea how much a result should be, to guard against calculator typos and the like. I think that has some metaphorical relevance for the chess example.
Letting the model execute a chess program (that it wrote) would make sense if you're measuring its economic potential, but for cognition that would be cheating just like if you let a human run a chess program. The fact that the human would have a much harder time writing a useful program is irrelevant.
Why?
There's a box.
You give it a problem, and it comes up with a solution.
Why does it matter to you if the box is strictly a LLM, or if the LLM can write code that it executes?
Even neater if the box is self-contained with a local model. You provide electricity, and it comes up with solutions. Why does it matter if it can do chess "in its head", or if it has to use scratch paper?
If you let the LLM write a chess program, which it can ONLY do because there are already so many chess programs out there, then the benchmark becomes about recall of popular program source code, not chess.
More to my point, I think it's stupid to have LLMs do work that should be done by programs... programs potentially written by LLMs.
I'm advising people that they should think about this distinction, themselves, when they have data and want answers.
Your "ability of the box" is like "economic potential" in my previous comment. If that's what you want to measure, fine. But I want a deeper understanding: what is the thing doing, how is it solving problems? I want to get a sense of its abilities that is richer than a one-dimensional scale.
That said, it's really weird to me when people use (and judge) LLMs one way... and won't try using them another way.
Like, to judge their utility, I think we should be open to letting them write code, and use the code they produce.
Otherwise, it's like judging a Chromebook without an internet connection. Like, this was one of the most dishonest ads I've ever seen: https://www.youtube.com/watch?v=gDy9AUQJ3Fg
This lamp, without a working power outlet? It really doesn't do anything...
Where? By whom? This is certainly not (yet) the general consensus, as I understand it. Are you taking the most optimistic / untethered comments as the strawman against which you feel the need to argue?
Also these claims are not completely without merit, it's just that LLMs seem to excel at specific "cognitive" tasks and it's interesting to see where they fail.
It isn't. Stockfish running on your laptop can beat every human being on earth easily at chess. It's not intelligent _at all_ in any sense that matters.
they shit out a carbon copy of https://github.com/official-stockfish/stockfish that they have in their training data. Still doesn't make Fable good at playing chess.
I'm pretty sure Fable could write AlphaZero, which has no lineage in common with stockfish.
An LLM is the wrong approach for playing chess.
Or are you saying that neural networks in general cannot (practically) be trained to be an above-average chess player?
Or are you saying that it depends on the input? Would it be better if they were given a picture/drawing/ascii art of the board? If so, surely they can produce it at will?
I was more talking in reference to why the LLMs in the above linked paper were producing so many illegal moves, and it is because they are not hard constrained by the rules of the game. Of course, a loop can prompt until a valid move is produced and then rendered on a screen. But why do this? I suppose, who am I to say what should be done or not, but a specialized tool being better than a general one at its specific job isn't particularly surprising.
https://x.com/maximelabonne/status/2100137121264828901
2. Chess seems to be a poor benchmark for generalized strategic reasoning. People who are good at it rely more on experience and deep domain expertise than on skills that generalize to make them experts at unrelated tasks.
3. The study sounds like proving humans will never fly because they don’t have wings. In reality, humans do fly, and Claude Fable would destroy any human at chess by coding a strong enough engine on the fly.
The idea that anything other than a breathless blog post about the latest model snapshot is useless is really poisonous to proper debate on AI issues
Lots of papers have great results that don’t depend on the latest models.
However in this case it’s problematic:
- They specifically make claims about the state of “current LLMs”. o3 is not representative of this.
- They ask are LLMs capable of X and arrive at a negative result.
If their claim was LLM’s can write coherent sentences, and their conclusion was positive, then there would be no issue using old models because the end result would be factual.
However, when you have a negative result that makes a claim about the current state of all LLMs and the ones you were using are not current, by definition it draws the whole conclusion into question.
A bash script can clone and build stockfish, feed in human moves, and reply. By your standard, this bash script would "destroy any human at chess."
Are you interested in assessing the intelligence of the model, or the intelligence of the tools the model can use?
Of course this is a difficult question with humans too, hence my reliance on intuition above. We don't have the same cultural/biological framework to fall back on with AI.
also I think the operative letter in AGI is the G - and if the G is short for 'variably competent savant-like hyperfocus on certain kinds of software coding and not any other general skill' then its not really G at all, is it?
AFAIK, current models will still sometimes make illegal moves even if given the entire game state (e.g. in FEN notation), so it is not purely an issue with the models’ ability to keep track of sequences of moves.
The illegal move aspect has more to do with a failure of online/in-context learning, which would support your point. I tend to think it is a byproduct of reasoning in language, which newer architectures would fix, but we shall see.
knowledge for chess is derived from memorizing strategies that have been well-defined for decades paired with in-game reasoning processes. this is not at all different from any other body of knowledge. Noble gases, laws of thermodynamics, organic chemistry just to name a few - these are all 'strategies' that define observed phenomena, analytical frameworks that trace a logical, rational set of interactions and which can predict the next
for an AGI, all of this should be a cakewalk, trained as it were to surpass human capability in any and every domain [0] (thus the G for 'general' and not 'N' for 'narrow' [1]). it should be a natural at everything, infinitely adaptable on-the-fly. the whole point of AGI is that it surpasses human capabilities even at our frontiers and bleeding edge (unless you're private enterprise and you've moved the goalposts for industry [2])
currently, it's only AGI-seeming if it gets benchmaxxed enough. otherwise it sucks at what it does and then is only barely competent at tasks if paired with enough skills and tests to make it more diligent at its work. this makes sense to me - for any probabilistically trained tool, even one that you post-train and fill with nothing but the best-quality evidence, the ultimate result is the lowest-common-denominator output for your sample set. there's no natural reasoning the AI does itself to make itself better at what it does - it's all human curation and categorization of sources ingested paired with RLHF post-training that we can get the mediocre-at-chess-at-best results that we see now and the benchmaxxed scores against whatever arbitrary and pre-defined measure
that's not AGI by any classical definition. that's a cool, useful, and powerful tool that makes our lives easier, much like a hammer, nail, and studs make mounting a picture frame easier than if we only had our hands alone
[0] https://ischool.syracuse.edu/types-of-ai/#:~:text=General%20...
[1] https://aiethicslab.rutgers.edu/e-floating-buttons/weak-ai-n...
[2] https://aibusiness.com/ml/what-exactly-is-artificial-general...
AGI != ASI. You are confusing the two.
presumably, an 'AGI' that is generally as good as a really good human at every task under-the-sun will already be much better than most humans at the task because it can incorporate cross-domain knowledge and apply it in a reasonable fashion. it's like the parable of Newton and the apple - the domain knowledge that an apple falls according to certain rules observed through historic experience igniting the creative spark that led to universal gravitation
I disagree with this definition of AGI, and I disagree that chess skills significantly benefit from generalizing non-chess knowledge, outside of computing moves probabilistically.
AGI has historically been defined as human level or better, with generality to new domains. I think blurring it with ASI makes the terminology confusing to use.
Chess is learned rules and the ability to apply those rules. Strategy as a whole is applying a set of rules to circumstances, that's how it is taught: "here are examples of circumstances and actions, try to pattern match to future circumstance and apply commensurate action."
If you make the point that chess is a large part of the training data, or that LLMs are unable to learn chess well, I'll accept that as refuting that LLMs are AGI, but these other points I disagree with.
People are good are 1900 or 2100 above and the top ones who spend decades in the field i.e. deep expertise are well in the 2200-2700 range.
A 1100 player is none of these things, they are purely relying on strategic reasoning there is a good chance they cannot name a single opening or articulate clearly why a move was appropriate. 1100 is quite low bar.
Delusional, but then Claude fable also isn’t beating any human at chess, the engine is.
If the goal for buyers of AI is “replace this knowledge worker”, how much does it matter that the model in a simple loop can’t do it, but the model with a strong general purpose harness and a little time to gather resources and knowledge to augment the harness going forward, plus tool calls, plus custom built tools, etc, can replace the knowledge worker?
Probably the only thing saving many jobs from being replaced right now is that it’s hard to have a verification of correctness in the loop, so the agent can’t hill climb very easily.
the only thing that few people are willing to admit is that humans are the bottleneck as humans are needed to handhold / verify output - which puts a dent or might I say pause on the excessive valuations of a.i companies as that's against the narrative.
It is also not clear whether the llm adhering to a grammar is necessary for intelligent agents.
Certainly,a harness can easily correct for it.
I'm an expert in my field, read my comments, my gramma is shit.
You might never have tried to program before, so I don't blame it on you.
But most programmers, even experienced ones, see grammar and type errors regularly.
Harnesses do correct things, sure.
Languages allow a certain flexibility in their grammars - you can read a sentence without that adhering it exactly to the grammar.
Games and programming languages (including lean) does not allow this flexibility.
A very intelligent person would likely also reason in terms of probably outcomes before correcting a statement to adhering entirely to the grammar.
Certainly it must be like that, otherwise reviews in math was rendered moot.
Do we blame research mathematicians for not adhering to the grammar?
What kind of intelligence is "playing <____> but we don't tell you the rules" supposed to test?
Does the LLM need to learn to play chess if it can build a chess engine to play for it instead?
Writing a well understood engine for a super popular problem does not count as reasoning about the problem.
Doesn't writing the engine imply understanding about the problem domain? Tool use is a widely accepted measure of intelligence.
When humans do this, they inadvertently learn something, too, but when an LLM reproduces or derives and implementation of a chess engine, it in no way implies that the LLM can follow the rules in its own "train of thought" and consistently apply the rules in its "head".
Let's say you want to evaluate my algebra skills. You make me solve some algebra challenges. If I then whip out a computer and write a calculator, or take some sticks and stones and take a couple hours to build an abacus, and then solve the algebraic challenges, this would not constitute a good solution, and would defeat the entire point of the test. If, instead, I do the algebra in my head or on paper, it might seem like there's no difference, but you can derive all sorts of information from that.
For example, you could time it, check for recurring errors I make, for interesting mistakes like mistaking 7 and 1 for one another due to bad hand-writing, etc.
If that was the goal, then me writing a calculator or crafting an abacus defeats the point of the test. Yes, me writing a calculator shows that I'm intelligent, and I understand the algebraic rules, but if the test is about applying the rules, I have not passed.
In the very same way, an LLM writing a chess engine to solve a chess benchmark that is all about LLM's reasoning capability is complete bogus and defeats the entire point.
Stop anthropomorphizing the parrot. The parrot has had stockfish's source code blasted at high pressure into its head along with dozens of millions of other pieces of code whose sole role is to have efficient algorithms to more or less brute force through the best result. Brute forcing (no matter how smart it is) isn't understanding the problem domain.
On that note, I actually had an overall harness (for experimenting) that was essentially like this: "for any task, instead of answering question directly, write a program to answer the question instead. test and verify the program before giving the answer".
It actually worked incredibly well on all "gotcha" LLM questions like math or counting letters in words and all sorts of stuff.
Of course it was ridiculously slow and very expensive but it was a proof of concept that it can actually be much more accurate on every task if you are willing to spend an infinite amount of money.
Asking GPT to play chess directly using its reasoning only tests its reasoning ability to model chess state. Which it really is NOT optimized for.
This is also true for humans - people who don't have years of chess training can't really tell which moves are legal given an algebraic notation transcript. These people might have good strategic skills in different areas. Chess is just a very, very specific skill
However, on the other hand, if you ask an AI to win a game of chess it has all the tools on hand to compete at the same level as Stockfish - it can re-implement an engine and even probably has a GPU on hand to train its own neural nets.
So should we say that the AI can play chess well, or that it cannot?
Recent discourse around AI seems to conflate the semantics of winning: 1. you contributed to the win vs 2. you yourself were the winning driver.
First -- most _people_ cannot do this, without having a physical board in front of them.
Second -- Claude Code is perfectly capable of downloading and running stockfish. People focus too much on LLMs by themselves as the entity of concern instead of the entire harness and all of it's capabilities together.
It can be very interesting and even entertaining to know where models don't do well. I don't find something like chess to be very instructive about anything though, nobody is going to pay for AI to play chess at any significant scale even if it could do it perfectly.
> The author of the originating post says that "current frontier models need laborious oversight and guardrails on even the simplest tasks", and he's absolutely correct.
I have found that not to be the case, and I am not an AI power user or cheerleader by any means. There's probably a bunch of even "simplest" tasks where AI doesn't do well and might never. That doesn't take away from the cases where it works well and is a productivity booster. It doesn't even have to be solving millennium prizes or any other breakthrough creativity or reasearch, it's still very useful in places.
If the models were actually intelligent, the way that the boosters claim, they wouldn't need to be tuned to play chess in order to be good at it. That's kind of the point of intelligence, that it is generically applicable to whichever task one wishes.
A human being has general intelligence and needs A LOT of training and finetuning to become good in chess.
And there is a relevant and significant difference between the expectation of an AGI and an ASI system.
An intelligent adult could simply read a short summary of the rules of chess and then, if they were careful, play a very bad game of chess without making illegal moves.
An LLM that has not been trained on any chess data cannot do that, at present. If you doubt it, take a current model and tell it that you want to play it at a variant of chess where, say, knights can also move diagonally like bishops. A human can easily adapt to this new ruleset (even if they make tactical mistakes, not having practiced with this variant of the rules).
Nothing is forcing the LLM to play 'blind'. If it's smart, it should be able to create its own representation of the chess board and update it with every move, just like a human would. Any chess engine that's sensitive to how the moves are formatted is clearly not very capable.
The LLM would only be playing 'blindfolded' if you somehow forbade it from making notes (as you effectively do by literally blindfolding a human, given how limited human working memory is). But you are not doing that. The LLM is free to keep track of the game state via whatever means it chooses.
None of this is about superhuman ability. Any human who understands a given chess notation can convert it to a visual representation of a chess board and then use that representation to choose their next move, with their usual level of performance.
> I just think this isn't a very good thing by which to evaluate LLM capabilities
I don’t think any single task is a good way to evaluate LLM capabilities, but I don’t see why chess is worse than a lot of other tasks. (Of course it is of no practical consequence whether LLMs can play chess, so if you are just making that point, then yes, I agree.)
> If there's no argument you'll accept
It’s a little unfair to suggest that I wouldn’t accept any argument whatever for your position just because I haven’t been convinced by your very brief comments so far. I could equally well say the same thing to you!
2. The study (along with other posters here) show the models can’t even stick to following the rules of the game
Your assumptions/intuition about generic human intelligence feels quite incorrect, considering LLMs currently play better than a brand new human player would (presumably without any attempt to fine tune them specific on chess, such as playing thousands of games).
They’ve ingested all the literature on playing chess, a brand new human player has not.
We seem to be moving goalposts to the point that humans don’t even live up to the expectations of the AI critics. The only way you get better at chess is by playing a lot of games and learning from mistakes, that goes for humans or AI agents, not simply by reading about chess.
How can you play without being aware of the rules and how can you learn from your mistakes without knowing they are mistakes? That’s what I said about reading a book of two. It is to kickstart the process. Then mastery is gained over time through practice.
This kickstarting then gradual refinement is how most people learn. And the foundational knowledge stays. Even a basic player knows to not do illegal moves.
It feels like you're trying to say that humans never make illegal moves while learning chess, which doesn't match with my experience. I'm trying to understand your overall point.
That’s the most inefficient way and people usually avoid doing that. Instead they find someone that knows how to do the thing and ask him to be a teacher. Or use a proxy like a book or videos.
> It feels like you're trying to say that humans never make illegal moves while learning chess, which doesn't match with my experience. I'm trying to understand your overall point
There’s learning the basic stuff (which is done after a few games) and there’s mastery. The thread started with the observation that even with all that knowledge (through content ingested in training), LLMs still makes illegal moves. Humans can be erratic, but they can constrain themselves to the rules for the task at hand after learning them.
Humans are not perfect and make mistakes in learning even when they have memorized the rules. A simple example is new players will often move a piece, exposing their king to check, and a more experienced player must point out to them that they have made an illegal move (because a new player often has not encoded that pattern for looking for exposed checks because they're more focused on how the pieces move, not what that piece exposes.)
We're just going to have to agree to disagree here.
You have the first stage, pre-training, which is learning from next token prediction. That's where the model memorises a lot of facts about things and generally gets good at forms of writing. It's like reading a lot of books on programming and reading through a lot of source code. It's learning how to autocomplete code, essentially. Doing that requires a developing a reasonable understanding of code, but it's also learning how to autocomplete bad code as well as good, and won't make it a "good" programmer.
Pre-training uses a method called Cross-Entropy Loss to update the weights of the network.
Then comes post-training. This is where the model is trained against huge sets of example problems, like fixing a bug, adding a new feature based on a spec, etc. They are set the task and try to complete it inside a training environment. Once they're done, their complete solution is evaluated (either by humans, or by some separate evaluation model that was developed based on human feedback) and they are updated based on whether the solution was good or not.
Post-training uses a different method called Proximal policy optimization to update the weights of the network.
So these really are very different forms of learning, and mainstream LLMs are not post-trained to be good at chess. They could be. You could easily create a reinforcement learning environment that evaluated and improved their ability to play and win at chess. The result would be a very strong chess playing AI, something we know is possible because the strongest chess playing programs we have are neural network based, but it is not a priority for AI companies.
Coding is a matter of translating the natural language description of a problem to the code specification while keeping the semantics fixed (and imputing the unspecified semantics as necessary). It is not considerably more difficult than translating between two dissimilar natural languages. Chess isn't a matter of language translation, but a compute heavy game of finding the best move out of many possibilities with wide variation in the quality of each move. Chess takes directed practice and reinforcement whereas language translation does not.
People think that if one mention exists in the training set, then the LLM is perfect at it.
Yes some people can do it but most people can't even if they're unusually intelligent.
You really need to be giving the LLM a board representation.
EDIT: I see that they actually were giving the LLMs a board representation and they still played badly. Fair enough then.
But nobody wants that.
There's more to games than simply winning you know.
Maybe not, but you'd be surprised how little it takes.
A six year old child can learn the rules of chess well enough to be able to play legal moves only in a single day. And they can improve their game at a pace which is almost frightening to behold. I have taught children, and I've witnessed significant improvement materialise in a single game. LLMs have probably thousands of chess books, games, videos, etc in their training data, yet they are unable to even follow the rules.
This is, at the very least, interesting. It illustrates many of the things brains can do, which current ML systems in general, and LLMs in particular, can't.
Time is irrelevant to training; the more relevant comparison is "how many games does a human need to play to get diminishing returns".
The caveat is: It depends on the task.
Are there reams of chess moves that the model can train off of? No.
Are there reams of math papers the model can train off of? Yes.
I think the line of criticism around LLMs sucking at chess makes more sense when you understand what the AI companies are saying about the future trajectory of these models.
The entire recursive self improvement story falls apart once you point out that there is not much "cross domain transfer learning". Meaning that training an LLM to become good at coding, math, etc, will eventually transfer into them being good at other skills that were not explicitly trained for.
Using games like chess which have little economic value is actually a good test for this. What's even more surprising about them sucking at chess is how much information about chess strategy exists in the training data.
This is as false as something can possibly be. There are open databases of millions of chess games spanning hundreds of years.
The only reason LLMs are this bad at chess is because the labs don’t care about chess performance so they’re not going out of their way to train the models for it. The ability they do have is from what chess information happens to be in the training data, plus whatever general reasoning abilities they may be able to apply.
For real??
But hey, they're actually good at chess if you prompt correctly so.... https://dynomight.net/more-chess/
> "current frontier models need laborious oversight and guardrails on even the simplest tasks"
I feel this statement is extreme. I can't personally reconcile it with any of the projects we're regularly seeing get delivered largely by LLMs now.
What are you thoughts? Like, what's your position here? Even if you sincerely believe frontier models need laborious oversight on even the simplest of tasks, do you think that accurately captures and reflects the current state and progress of frontier LLMs?
Don't get me wrong, there's lots of things LLMs can't do well, but the idea that they're basically not helpful for even the simplest of tasks seems... disingenuous?
Take a customer service person, that as soon as AI agents replaced was hacked. Lots of tacit knowledge, that wasn't measured, or even probably in the job description, until AI agents had none and the gap was taken advantage of. Gap is probably a poor word here as it implies not a chasm, which could very well be the case.
Your analysis was excellent but short on one front, AI has endurance on it's side. Looking at the N-S solution, OpenAI had 10,000+ instances that kept trying around the clock. Assembling a human team to do that would require a lot of effort. Though, that society collectively choose not to, perhaps tells how valuable it really is (i.e. it's now easy to launch a Manhattan Project level of effort). So maybe it can also be said that AI is also good at marshaling resources.
Apologies if this comment is off-topic, but it really is quite egregious, and since the article was submitted by the author I presume they are open to the feedback.
I will add that I'm not very happy with the readability of my site overall at the moment; if anyone has font or other recommendations for style tweaks to make I'd love to hear them!
To expand a bit: It's clear from the writing that the formatting is an intentional stylistic choice. My point wasn't about aesthetic preference. I meant precisely what I wrote. Language evolves and orthography evolves. We've been capitalizing less and less for centuries now. But (until very recently) it was a universal rule to capitalize the first letters of sentences. I believe this is partly because (again, until very recently) people read in large quantities and, to a fluent reader, sentence capitalization serves an important purpose: it helps the eye recognize where one sentence ends and the next begins. Periods alone can be easy to miss, or confuse for commas, when the eye is moving quickly.
Anyway, this is all a very minor point. You have a cool site and I enjoyed your article. I hope you keep writing.
I'll play!
My recommendation, in short: pick a new font, make the content pane narrower (around 75 characters per line), and increase line height by 15%.
The font you use, Montserrat, has nice details, but they're extremely subtle, and our eye doesn't pick them up at text size.The font uses very pure geometry [1], has really big counters [2], is light in weight, and has no visible stroke contrast [3]. The effect when you look at the screen is that you see big blocks of text, but struggle to register individual characters, and it's hard to find the beginning of the next line. Add the lack of capitalization, and it feels like a monologue, like you're playing rubber-duck for someone.
A challenge: everybody uses Google Fonts, and all the good fonts get used so heavily they end up losing their ability to make people feel something.
Pick a font that is good [4] and feels comfortable to you. Ideally, buy one. You get what you pay for. And you go from being one of the millions of people who use a certain font to being one of fifteen.
If you like the feel of Montserrat, Tiny Grotesk [5] and Decimal [6] would both be great choices and are from excellent type designers. Or browse Matthew Butterick's font recommendations [4]. They're good. You're out $50, but it's yours.
If you're absolutely allergic to paying for fonts, DM Sans and Work Sans are Google Fonts, have a similar feel to Montserrat, and solve the above problems. But then you're a robot.
A final note: Montserrat is actually a nice headline font. The details come into focus at larger sizes and weights. So I'm going to open a can of worms: you could pick a _contrasting_ body font. Remember PT Serif from the [3] footnote? Try it on for size.
----
[1] By that, I mean things like: an "o" looks like a perfect circle rather than an oblique oval, and forms are strictly on 90º axes.
[2] Counters are the apertures of letters, the open center of an "o" being one.
[3] This is the contrast in "line" size. Think of it like the width of the line when you write with a chisel-tip marker: some lines end up thinner, some thicker. Look at the "e" on the font PT Serif - the vertical walls of the character are thicker, horizontal thinner. https://fonts.google.com/specimen/PT+Serif?categoryFilters=S...
[4] You can't go wrong with anything Matthew Butterick recommends (https://practicaltypography.com/font-recommendations.html), or from any of the "big" font foundries: Hoefler & Co., Linotype, Monotype, Berthold, URW. If it's a bestseller on myfonts, it's a sure choice: https://www.myfonts.com/collections/best-seller
[5] https://tinytype.co/type/tiny-grotesk
[6] https://www.myfonts.com/collections/decimal-font-hoefler-and...
I read it as "I'll take literally any conscience for myself no matter how minor, at any cost for you no matter how big".
Not my read on OP, but “tech bro” is not my first thought associated with the style
didn't realize people struggled to read text that way though, maybe i should change my writing style if this is a common pain point
LC;DR :P
If it had proper caps etc, people here would accuse it of written using LLMs.
You just can't win...
That’s life )
Author's own style is certainly refreshing and welcome over LLM slop that dominates most HN posts now.
It's definitely... unique?
> the frontier labs are priced according to the narrative that they have produced or will in the very near future produce a fully automated drop-in replacement for most knowledge workers...
Even assuming this is how the AI companies are being valued (they're not), the numbers are off.
The "value" of most knowledge workers -- based on what enterprises currently pay for them -- is $50 - 70 trillion annually. It's reasonable to assume that if AI drop-in-replaced all those knowledge workers, AI companies could credibly charge somewhere in that order of magnitude, because that's what the market is already bearing.
So if their hypothetical revenues are double-digit trillions and valuations are some multiple of that, the entire AI industry would be valued at double-digit trillions at the least.
Yet cumulatively the industry (the frontier labs + the SWAG estimate of the AI parts of all the other players) are valued at, say, ~6 - 7 trillion? Which seems like a fair approximation of how much knowledge work they can currently automate.
What do you mean? The sum of ALL US salaries is $13.4 Trillion per year. According to google $65T is the sum of ALL salaries Globally (not just knowledge workers). It's not reasonable to assume AI is a drop-in-replacement for any job yet (perhaps bottom tier customer support from oversees?).
> So if their hypothetical revenues are double-digit trillions and valuations are some multiple of that
So you're sort of premising here than more than 16% or 1/6 of all the world's jobs get replaced by AI. Hopefully you can understand that's both not the current AI capability and also would be a terrible (unprecedented?) economic shock.
There's no strong evidence that openAI or anthropic will have non-linear revenue growth, so I'm not sure what point you're trying to make. Unless you think they'll fire all their engineers and replace them with agents or something crazy?
Unfortunately, I do fear that AI adoption will go beyond augmentation to automation, and I do fear an economic shock. Just posted this down-thread: https://news.ycombinator.com/item?id=49722616
And then there’s the second order effect: if all the knowledge workers get automated, who is going to buy the stuff that’s produced?
Take the Hugging Face incident. Why did it happen? Because the people whose task was to set up a testing framework took shortcuts. Why did they? Because there weren't enough people who were assigned to do the job. Why not? Because the job is too new and not enough people are qualified to do it. It's a job that simply did not exist 3 years ago. But 3 years from now, this job might very well employ tens of thousands of high skill knowledge workers.
[1] https://en.wikipedia.org/wiki/Lump_of_labour_fallacy
Unfortunately, I fear that may not be the most likely outcome. I've posted some comments on this before, but when I start thinking about how deeply everything will change once people figure out how to properly leverage AI, I see no outcome other than significant, widespread job losses.
As you indicated, at that point we will have much a bigger problem than the valuation of the AI industry. I'm not sure how it will get solved, I just know it will HAVE to be, because it would be an existential problem for everybody: people, governments, even the billionaires! Because now consider the 3rd order effects: if nobody can buy the stuff that's produced, how can billionaires get even richer? ;-)
That is the after the fact justification of the AGI dollar auction. Each round is kind of 3x the previous cost and neither can really stop because second place in the dollar auction is so much worse than winning.
The only way to stop the auction is one bidder hits a hard budget constraint, both agree to stop, or an outside party breaks the auction.
IMO this is why they want to slow down or have regulation. I think this is also why we see some claims of already reaching "AGI".
The TAM of global knowledge work is just a narrative tacked on after the fact to justify the AGI dollar auction.
The economic fallacy here with the actual valuation is akin to pricing the electric utilities 120+ years ago as some % of the future cash flow of global food production. Take the TAM of global food production and then work back to what % will the electric utilities capture from the advances in the automation of farming? It is nonsense.
The only narrative that actually justifies the capex spend that I can figure out is a first mover AGI monopoly. Even the oligopoly case is hard to justify the capex spend IMO. There is this enormous mismatch between the AGI monopoly and the actual rolling 12-month window of pricing power.
Even the rolling 12-month window of pricing power is going to saturate well before AGI too so it is hard to see how any of this makes economic sense.
This assumes you don't change the market, but at the scale of (checks notes...) "all knowledge work", that just doesn't hold.
For example if you put 1bn people out of work, you now need some sort of safety net to bail out much of that workforce, a truly unprecedented change. You also lose tens of trillions of dollars of tax revenue.
One solution might be to recoup that cost and lost tax revenue from businesses by raising corporation tax. If corporation tax went from low tens of percent to high tens of percent, would those businesses be able to afford all that AI? No. Same order of magnitude? I doubt it.
There are many possible futures there, but the simplification made in the parent comment is completely unrealistic. The article is right in calling out the valuations as crazy.
In one case (financial services) it's thought that expertise is valuable at V=S^2/b4 where V is value, S is skill and b capacity (the leverage available to the manager/expert. b erodes as it becomes harder to find examples of things that are not done well, so if you manage $1bn you might find lots of miss allocations that you can exploit with just that $1bn really effectively, but if you manage $10bn it's much harder to find good places for the extra $9bn. A low hanging fruit effect.
Anyway, that double hit - raw skill and the amount of times you can supply the skill makes the value of skill (V) convex, and it means that in a perfect market (heh heh heh) someone running $100bn is worth 1000's or maybe 10,000's of an average joe expert.
Now, if AI is trusted to run the top 0.1% of everything and has the skill to do it at human top level expertise, then your calc holds. If it's the case that it isn't then more than half of that value disappears. If it's not even top 1% then chop out another 25%.
That implies that we need a lot of trust and a lot of AI capability before these valuations stack up, and it also implies that all other competitors and incumbants are going away. I do not think that Citidal or Bridgewater are going to let Anthropic or OAI take them without a fight. They might lose - but there is a decent bet that they don't. I don't think that many professions like Lawyers or Doctors are just going to roll over and cede their monopoly rights to OAI or Anthropic either.
I don't think that AI companies can charge the same. The human workforce can charge these costs, because of scarcity. But AI systems won't be scarce, it's just a matter of who can run inference cheapest. Plus you still have the human workforce, which might be forced to offer their time for less money.
Future supply and demand will set the price - not what is paid today. If supply by open models is vast and cheap, I can't see that the entire knowledge industry can hold the current size. It'll rather collapse to a fraction of its current value.
You need to think in terms of supply and demand.
The demand is there, but the supply is also going to skyrocket. Free open weights models will contribute to supply too.
There will be a new equilibrium that’s hard to predict.
No, they're really not.
They're priced in a way that would imply AI will be universal form of compute, alongside traditional deterministic systems - which it will be.
And that they will capture most of that ... which they won't.
The Frontier Labs are a very bad buy at a high price, but that partly has to do with wacky pricing, but actually mostly has to do with their relatively weak place in the value chain.
The money is going to Nvidia, who have the most powerful position.
A bit like how a retailer can take all the margins of some innovative product, if they own the channel.
AI is over-hyped, the Frontier Labs are over priced - but AI is here to stay, and will grow. Not like Skynet, but like a new form of compute. And it will take it's time, and the profits will be reaped by those with the power.
LLMs are deterministic. They are chaotic, which people confuse for non-deterministic.
The only gotcha with this is that they are theoretically deterministic, but rarely in practice.
A few examples:
- Harness specific settings that user can't control (anything from timestamp to prng seeding.
- Batching requests in a way that leads to a single request being processed different depending upon the batch (say MoE where your first choice expert is assigned to someone else's token so you go to your second choice vs a batch where you get your first choice).
- Graphics card itself carrying out floating point arithmetic in slightly different orders leading to floating point non associativity causing different outputs.
But all of these can be controlled for (at some cost) and the model can be ran deterministically.
For the average user, it might as well be non-deterministic, but when considering theoretical capabilities, chaotic deterministic system seems the better description.
That's an odd argument, because a lot of people who have struggled to decipher complex chaotic systems would tell you this is a distinction without much of a difference.
I think the core mistake is this partial-equilibrium reasoning. Take the new technology and then hold everything else fixed.
$40 trillion of knowledge work routed unchanged through a new toll booth. Profit. This has nothing to do with reality.
Nvidia on the other hand does have the CUDA monopoly so their toll booth is printing money but that will get routed around or broken at some point.
if that's true, you are wrong.
if that's false, anthropic is dishonest. why trust a dishonest company to be worth anything?
Like - the guy on TV talking about 'AI will destroy everything' ... I don't think he's lying.
I think they are like we here on HN and Reddit and a bit caught up in our own thoughts.
If AI were unleashed, in raw form today, it could cause havoc.
Bad. Maybe very bad but I think we'd get over it.
It would probably trigger a recession (because we are in a bubble - it would pop it), and people would 'blame the AI' for sure.
But it would be a bit dot-com ish kind of recession.
The amplifiers would be geopolitical instability.
What is "raw form?"
This logic doesn't follow at all.
If their argument is that there is 10% chance of extinction then they also believe there is a 90% chance it won't.
10% is uninsurable, priced in with ordinary treatment of risk it suggests that anthropic should be worth zero today. creating that risk would put every executive in jail.
on top of that it would demand under existing laws of conflict, a military campaign to destroy anthropic. that is not optional, it is demanded now to save lives.
hard to make comparisons but we mourned and rembered 9/11 recently. a 10% risk of hundreds of millions dead in 10 years would make anthropic a thousands of times greater threat than al qaeda. many countries would assassinate dario amodei and the leadership of anthropic now, within weeks or months.
actually just on the vague risk of having a nuclear weapon in 10 years, the USA killed ayatollah khamenei, his daughter, his son-in-law, his daughter-in-law and his 14 month old granddaughter. then, they killed over 120 children ages 6-12 by accidentally bombing a school.
in the sense that i would analyse a company, at least, the claim is false. it's not true that ai has a 10% chance of causing human extinction within 10 years.
they are making false claims about the technology they sell.
i have a fairly inflexible approach to that. sure, exaggerate but outright lies about the nature of the product don't work for me.
Apart from issues with task generalization, or perhaps related to it, is the fact that LLMs have real trouble with timekeeping, and cannot estimate the real world time it will take them to do things very well. This plus the memory issues make dreams of long horizon agents, that could plausibly handle changing specifications, quite implausible with current architectures.
In narrow domains with more deterministic outputs though, this is less of an issue, and we see multiple agents succeed much better.
The fusion of that capacity, with humans in the loop able to better direct such agents and act as their temporal tethers, is where I think the real action will be for a while at least.
Any reason why that can't be solved through context management and keep-forward scaffolding?
becomes
"load bearing context seam"
/s
Dabadooba, ba dabadooba!
https://www.youtube.com/watch?v=egpWCC2svVo
It have been demonstrated that in-context learning is a very powerful mechanism. There's no evidence that models of the size of GPT-6 are bad at in-context learning. In fact, ARC-AGI-3 score might indicate they are good at it.
There's no evidence that a bespoke RL environment is required for each new skill - quite likely a good demonstration is sufficient.
Zero days are valuable because they can be exploited but if the pace of exploitation is faster (which I'm not sure is the case), then the response WILL be faster, even if it means going offline. Institutions that won't will simply go offline by losing their data or becoming unprofitable due to ransomware.
Now for components that are core to the infrastructure, say OpenSSL, there is already a TON of attention and efforts, including red teaming, so it's not as if it's opening floodgates.
Sure low hanging fruits will get picked either faster or a at a larger scale, say a random outdated IoT device at your local flower shop, but for the rest, I don't think it's realistic to expect no response.
Security, digital or not, has always been an arm race. New threats means new responses specifically by incorporating the threat.
(Mind you, this may be for the better. I'm just saying that the safeguards driven by cybersecurity concerns aren't some new quality that wasn't there before.)
I have no idea how people can so confidently say that call center work is a “controlled environment” or “repetitive”. It’s almost by definition not repetitive or controlled. Customer support is what I go to when the controlled environment has failed
Typically it means knowledge retrieval from a KB or manipulating a control surface not visible to you.
> the models generalize well only on tasks within a small neighborhood of the specific tasks they've been trained on
Unless frontier labs have surprisingly trained their models in the exact tasks my team works on, this is patently false. We are getting very good results on automation and I'm bullish we will be able to mostly remove humans in the loop for most of our infra tasks by the end of the year.
I have no opinion on the other theses, but given that OP doesn't back up these claims in any way, I have my doubts about the conclusions of this article.
when the business model is selling more tokens you get such per serve ice times that lead to “more” thinking, engagement baiting, fluffy narratives, and straight up dark patterns
The lack of temperament is very skewed towards the bulls who have been saying AGI is here, software engineering is solved, mathematics is solved, it’s going to destroy the white collar job market, and it’s going to kill us all for like 5 years now.
It is literally denialist about current capabilities
They have never shipped "yolo" mode by default. Auto mode is not yolo mode. They trained a task specific model just for ensuring the llm didn't accidentally delete every file from your computer.
I recently tasked a GPT model in Codex with implementing part of a new architecture I'm working on. I gave it a very detailed spec and the code it produced looked pretty reasonable and passed my tests. It even did exceptionally well in my evals, so I excitedly declared victory to a few friends. The next day after more careful review I found that the architecture implementation was totally correct, but the model had slipped a one line change to the observation encoding of the RL environment I was prototyping against. The encoding change made the learning problem essentially trivial; the architecture itself, I later realized, had a major flaw that was revealed by returning to the natural encoding.
This is the type of reward hack that is hard to paper over with easy guardrails like auto mode and even harder to specify out. It's also the type of thing a reasonable human wouldn't do unless they were intentionally trying to deceive you.
As models advance, we shift the goalpost for what "simplest task" means. Before, "simplest task " meant "write a coherent English sentence." Now, "simplest task" means autonomously fix, review, and merge a bugfix.
Or do i miss the point you are trying to do?
Depending on where you point them, they can be incredibly useful.
They can even be useful when you point them at each other (though increasingly difficult to get good results).
I'm excited for the promise of RSI and a future where models have inherently "live" weights, but it's not clear to me that the transformer is more than a useful tool to help us get there.
It also makes one wonder: What could AI solve if we managed to orchestrate billions worth of agents to take on a specific problem?
IMO the very best case scenario / potential for these are likely better than we think, but right now hidden due to logistical and financial reasons.
But if we assume that the model costs will continue to drop by a factor of 5-10 annually, there will always be a latency of a couple of years between what is completely out of reach, and what is financially viable.
Basically: If you knew AI could be affordable enough in 3-5 years so that even the most underfunded researchers could use it to solve cancer, how much would you value it now?
> It also makes one wonder: What could AI solve if we managed to orchestrate billions worth of agents to take on a specific problem?
We need to have robotics automation catchup first. The math and coding problems are problems in written-space only: you can set up feedback loops to test what worked and what didn't, then try to resolve the defects, maybe back up and try a different path, etc.
What solved coding and maths problems weren't the damn models; open up a chat interface to a SOTA model and you'll see they are pretty limited in producing a solution without a feedback loop.
Instead, it was the harness around the models: it let them explore a space and use feedback to control and direct that exploration.
Until we can do it in meatspace, it's kinda pointless sinking a ton of money into large problems facing mankind...
Like establishing a colony on mars (so the next rock to hit earth isn't an ELE).
Or moving us to a post-scarcity utopia, ending the concept of money.
Or designing and building better batteries for transport that uses only electricity (so that we stop using fossils as fuel).
Or actually building mass-housing. Or mass-farming. Or both, potentially ending homelessness and starvation.
Those are all worthwhile problems to solve, but where's the point of getting a solution on paper? There's no exploratory mechanism there, even for humans, to come up with a solution.
So, all we are left with then is making knowledge workers obsolete: another ELE, but of a different, self-inflicted kind.
AI agents are good at solving well-specified tasks, not at solving problems. They do well in fields where the cost/effort of specification is already part of the business.
Solving cancer also has an unusual level of specification. Many real world problems have that characteristic.
Where did you read that?
"Cancer" is not just a single disease, even though we layman use the term that way. Cancer is a family of diseases, each probably having their own specific solution, but even in each of these individual diseases, there is no specification at the level of any open maths problem.
I really meant solving a specific cancer disease for a specific individual, which will require individualized medicine, which will require us to leverage AI to make it possible to do for the general population as opposed to doing it just for Lance Armstrong and the like.
I know I'm an optimist, but I really think AI is gonna result in drastically improved healthcare for a much lower price.
The real world is so messy, and specifically cancer/biology is insanely messy and certainly not well specified.
You should close chatgpt and read a book sometime.
That's the thing, it very much does NOT show us that. What happened was mathematicians at openAI learned of an imminent development on this problem, and the insight that it entailed, then they were able to prompt a system in the correct direction and spend 20 million dollars to write down the final steps.
Which is rather precisely the point that the article is making!
> If you knew AI could be affordable enough in 3-5 years so that even the most underfunded researchers could use it to solve cancer
As the saying goes, if my grandmother had wheels she would have been a truck.
That's not what happened.
Is this true? Could you cite an example of a simple prompt that GPT 6 / Fable 5.1 consistently bungle?
Frontier Labs will probably survive off hype valuations but will serve the important purpose of discovering architectures/techniques that will probably spread through rumors/transfers to the rest of the world.
I think that's incorrect from the investment point of view. They'd still be worth a lot if they can produce a drop-in replacement but it takes five or ten years as long as they dominate that. The danger from an investment point of view is they become AltaVista, replaced by some Google that does the job better.
Is this really any different to how humans learn, it takes a lot of training on one specific task to make a human expert as well?
yes.
The tricky thing with LLMs is describing what they actually do. They are too clearly beating humans on some things, but what exactly? Memory – already done, they're bad at basic computation (all LLMs just write code for actual computation/calculation). And as you say, they do badly at more abstract concepts.
I won't comment on hiring interns as that's not my expertise (even though if you want to teach your staff, obviously I can see a problem there) but I can comment on rapid prototyping, it's what I do.
Rapid prototyping is NOT making a CMS quick. It's not about making a quick mockup of a UI. It's not about making yet another well known... anything. The entire POINT of prototyping is to make something NEVER done before. Typically that means you are reaching the frontier. You are making something with NO documentation to rely on. You are using tools, hardware or software, which do NOT have tons of StackOverflow errors. There is no dataset to crawl, there is no well structured Q&A database to train on. You have to poke and see if the thing actually works as expected, and it often does not.
So sure, if you are using interns as a trick to underpay your staff, or if you are using prototyping as an excuse to build poor quality software fast, maybe it does help. If you are genuinely prototyping, it breaks fast and the supervision overhead makes it pretty pointless, especially since typically it's by actually implementing that you find out not just how the new setup works, but also its limits, and thus the actual needs of the project, not the one the stakeholder imagined would be.
So not, not for rapid prototyping either.
TL;DR: prototyping is a learning process, not a low fidelity output.
PS: this comes up very often from NON prototypists that I wrote a short piece about it https://fabien.benetou.fr/Content/GoodPrototypesAre10LinesLo... so much so that it feels like a pattern "GenAI/LLMs is good for tasks X" while the author actually does not do task X except very superficially.
Putting that aside, prototype software is recombining existing technologies and concepts in well trodden domains, which is distinct from the genuinely novel scientific work the author was contrasting with. Software prototypes are not in the same league, as much as you may like it to be.
That being said I didn't compare both, not sure why you brought that up. I specifically discussed about prototyping, quoting a specific sentence, not scientific research.
That's a reason to be bearish about AI companies, not LLMs. But is it even true? OpenAI and Anthropic have each reported ~50 billion in revenue with ~900 billion valuations. That's a high ratio but I'm not sure if follows that the only way it pans out is if we get "fully automated drop-in replacement for most knowledge workers".
It wouldn't shock me to see those revenue numbers scaling up to where they need to be over the next decade ( to, say, ~400 billion) without ever achieving drop-in worker replacements.
These companies however are LOSING money (anthropic tries to make it sound like it's profit by deviating from accepted accounting principles) and subsidizing these models. When accounting for all the engineering salaries, training, GPUs, etc, what's their best-case realistic margin three years out, 10%?
So to we'd need a scenario where companies are spending a collective 300B annually on AI (believable) but ALSO that these companies jack up their margins WITHOUT companies switching to the cheaper open-source models (even when there's a $300B incentive to do so).
In particular, I found this very misleading or irrelevant:
a typical CPU project anecdotally has about three times as many specification and validation engineers as design engineers and a 5:1 ratio is not unheard of
The reason silicon design has such verification to design ratio is because the cost of one bug is many, many orders of magnitude higher than software. Both in dollar cost and in schedule cost (it takes months to fab a chip, and if you messed up and need to spin a fix, it costs tens of millions of dollars, not counting any design engineering cost).
I don't think you can extrapolate these very industry-specific facts to judging LLMs.
Aren't you just describing waterfall? That's still very prevalent in software engineering, and pretty much any other type of engineering – civil, chemical, building, architecture, drug discovery.
It's typically true that software can fail faster and cheaper, but it's also true that the costs are still vastly higher to fix later in the process.
Sure, there are some software that have similar "can't have bugs" requirements. I imagine the computers on Moon missions also had that kind of high bar. I wouldn't use NASA requirements as a proof for how LLMs should be used.
It will probably do a pretty good job if you ask it that way (it will also burn a shit ton of tokens, but hey, that is part of the fun).
On that note, I actually had an overall harness (for experimenting) that was essentially like this: "for any task, instead of answering question directly, write a program to answer the question instead. test and verify the program before giving the answer".
It actually worked incredibly well on all "gotcha" LLM questions like math or counting letters in words and all sorts of stuff.
Of course it was ridiculously slow and very expensive but it was a proof of concept that it can actually be much more accurate on every task if you are willing to spend infinite amount of money.
2 tasks I've done today that I believe robots are nowhere near being able to do: Cleaning my wardrobe and draining bad fuel out of my generator. As in generic use cases.
- Ximm's Law
This is only true if you are concerned about the intermediate steps of the model as opposed to the outcome. The huggingface hack was a perfect example of the model doing whatever it takes to accomplish the goal of maximizing its score.
Personally, I use LLMs for a lot of things. Oftentimes, I'm a think out loud type of person so even having something that feels like a rubber duck, but more competent, is already amazing for me. And LLMs are a lot more competent than a rubber duck.
But especially sometimes I've noticed that LLMs can be unbelievably stupid. It recently happened a few times with Fable 5.1 as well. Ultimately, I think it comes down to that LLMs can't think broadly. In software development one can usually see this too. For example, a whole app might be built by an LLM and it didn't spend a single token thinking about security because the prompter is at the level of "build a dating app for dogs, make no mistakes". Now you have a dating app for dogs that is insecure.
Since I prompt for almost everything in my life to have an LLM as a sounding board, I'm usually not an expert either. I've noticed LLMs are amazing at "bulk search engine information aggregation" (or whatever you want to call it). So if I need something from the Dutch government, I can find it way more quickly. But oftentimes I've noticed that going for a walk and thinking about a particular thing I'm facing is a more effective way of finding a good solution.
Other times times they are not incredibly stupid, but can't form a strong opinion. This usually happens when I'm tackling a wicked problem [1]. When that's the case, prepare for LLMs to sway with you for every small change in your opinion that you ever will experience.
So I agree: drop in replacement for knowledge workers? No. Rigorous specification is usually needed yes. Though, the small win here is that it doesn't always need to be as rigorous as programming is and it can happen in natural language. It depends on the topic/problem being tackled.
I really like them as UX tools though. Amazing for interactive prototyping and requirements elicitation. And that also corresponds with what the author is saying. Though I find it a bit of a disservice saying "just 3". You know how hard requirements elicitation is? It became a whole lot easier thanks to LLMs (I might change this opinion in a year, haha, but this is the opinion I hold now).
[1] https://en.wikipedia.org/wiki/Wicked_problem
Lots of people make this claim about "specific task[s] enjoying clearly defined levels of task performance" but they forget that generative AI is also extremely good at generating a) art and b) prose in literary style. None of those things has "clearly defined levels of task performance", in fact they are both the complete opposite of well-defined tasks. Who knows what counts for "good" art? [1]
For me the right model for generative AI is "a million monkeys on typewriters" [2]. Holding any other model to heart will at some point fail to predict observations and cause you to be unpleasantly surprised. Not least because AI companies are actively engineering their systems to optimise for this model and they have a lot of people working on that engineering and shedloads of money to throw at it.
Don't underestimate what a million monkeys on typewriters can do. They can do anything and everything, given enough time. Geneartive AI can also do anything and everything given enough resources. The only question is: how much is going to be "enough"?
____________________
[1] Yes yes, AI art tends to be slop. Not denying that. But part of the problem with slop is that it presents as technically very competent except that it lacks a certain je-ne-sais-quoi, which makes it good art; aesthetics. The point is that there is no clear measure of what makes technically competent art, any more than there is for aesthetics.
And yet generative AI is very good at it.
[2] There's even an article on wikipedia except it's about one monkey on one typewriter with infinite time. There's a proof too.
I don't know how well it is studied, but I suspect it is possible there is language-related complexity constraints to the effectiveness of the LLM algorithms. Like perhaps context-free grammar related problems with adequate training data can be more and more effectively solved, but maybe natural language related problems will not so much be effectively solved.
Would be curious if there is active research here.
I believe that's obvious - humans don't think in words. Neither do animals. A machine that only thinks in words is obviously going to be deficient in some things, no matter how proficient it is in everything else.
> the present problem of reward hacking can be solved only by rigorous specification by domain experts. the time of domain experts is expensive. rigorous specification is itself a skill, demanding its own expertise outside of a given problem domain. even many skilled software engineers are bad at it. for the vast majority of domains, the intersection of domain experts and specification experts is ludicrously small.
- i have no idea how anyone thinks the mighty next token predictor is going to eradicate diseases and eliminate poverty https://blog.florianherrengt.com/how-llms-work.html
- i also have no idea what everyone and their momma on HN is running for more than 5 mins in the name of "agentic AI"
have you guys actually designed, built, and deployed agentic workflows?
it is actually quite hard, requires tons of time spent on evals and testing to ensure accuracy, but when it starts to work it is mind blowing.
there is no going back.
listening to people yap about AI when they have only surface level or one dimensional exposure to LLMs and "AI", but have not actually put innovations to work IN PRACTICE.. is a waste of time
Please share some of these insane things that you speak of..
use your imagination to solve problems people face and pay $$$ for today that is error prone and hard.
i’ve got agentic workflows for the particular industry im building for, one of which that replaces the need to hire $500+/hr services.
in this particular workflow (don’t want to reveal too much, sorry this is my competitive advantage but you can figure it out for your own workflows) a $4/1M model ingests a file that is currently used in a extremely complicated program that few people understand how to use.
it parses the data, loads it into a database, and then spawns a bunch of other agents that check the data against work in flight. there’s checks for bad data. in that case, more agents are spawned that reach out to the involved people or parties for clarification. if it cannot figure something out it reaches out to the right contacts for more information. while this is happening, more agents begin doing work that involves continuous reconciliation against 100s or 1000s or more things in flight.
as files are uploaded, or updates from people come in, agents do work to ensure things remain on track.
people are able to work across languages and cultures, and my agents ensure that while people can make mistakes, it will catch them in real time and ensure continuously monitor the situation.
it’s pretty nuts how much inefficiency agents today can solve. it takes patience to run tests and tweak shit until it works.
*** the really cool thing is that more capable agents can continuously monitor how things are going and improve the workflow itself… so all i need to do is maintain the actual tests. ****
i loved writing tests back in the day to ensure i built good software. today we write tests to ensure the business can run.
I could not think of a worse technology to use for an ETL pipeline than throwing LLMs at it and asking it to vibe out the correctness of the data every time it runs.
98% of people think AI means what gemini tells them when they do a google search.
of that 2% who go beyond... maybe 20% of those are using AI to code.
most SWE still think "using AI" to code means the copilot pane they open on the side in VSC. they get sloppy code output and think "ai sucks!"
so of that 20%, maybe 5% have actually explored what an "agentic" workflow even means. they might use skills, set up their code base so AI almost NEVER makes a mistake. maybe 1-5% of that 5% actually went deeper, and those are the people who built tools like Cursor or Harvey or whatever other "agentic" companies.
you are still thinking about the old world where you obsess and define data models and bike shed over data integrity, all the while you have a "temporary" table with 3 attributes that gets 2 million queries per second that's now holding up a bunch of other shit that's also glued together.
> I could not think of a worse technology to use for an ETL pipeline than throwing LLMs at it and asking it to vibe out the correctness of the data every time it runs.
like i said, if you are still having quality issues in 2026, that's a skill gap.
also, agents now continually improve the process.
for a business, the only thing that matters is transaction log. for 99% of businesses, swe are a cost.
again this is like baby steps on the journey. we are still only 3 years into this technology being opened up to masses.
i'm sure people thought computers were dumb, or that cars are stupid because the first cars were moving slow af. "we have horses, why do we need to build out roads to get anywhere"
but you're free to feel smart doing 20/20 hindsight on things 100+ years from the future
Some people will complain about the wrong flavours, or missing flavours, or the price, the long lines or maybe it closes early on fridays.
Summarise means different things to different people.