I don’t know why you think this is true.

I mean you can just try it with DeepSeek or any other large model yourself. This is literally a solved problem now.

Absolutely not.

Evidently you need to read up on how reasoning chains work.

Not really. It might have similarities, but I would never say it’s the exact same problem.

It literally is the same problem. Your brains didn’t evolve to do formal logic natively. We emulate it exactly the way the LLM does.

But for different reasons. Transformers because they are state destroying. Humans because they have limited and volatile working memory.

No, for the same fundamental reasons. It’s got nothing to do with state being destroyed either. It has to do with the fact that stochastic systems aren’t a good fit for doing symbolic logic.

It’s certainly different because pen and paper are most often used to enhance working memory, but the fact-based reasoning and the algorithmic state machines are encoded in our brain which is impossible for a transformer LLM.

Fact based reasoning is something our brains are famously terrible at doing actually. That’s why we use tools like computers in the first place. Our brains can be trained to express patterns of formal logic, and an artificial neural network can be trained to do the same thing. That’s why modern LLMs can reliably tell you the number of R’s in strawberry.

It also requires understanding how things actually work.

It doesn’t, that’s the whole beauty of genetic algorithms. All you have to do is specify your selection pressures and your goal criteria, and the system evolves a solution to fit the shape your desire. The LLM doesn’t need to get better at doing math, the stochastic approach means it converges on a solution given the right environmental pressures. And that’s why hallucinations don’t matter, they get weeded out by the attempts being tested against the environment.

Which is the same way it gets better at coding and yet still can’t actually solve complex problems in design space, constantly has to use ensemble sampling, and constantly has to be told to re-roll the dice whenever the test fails. And that behavior is so costly under the hood that it’s eye watering.

I can tell you haven’t actually worked with these tools recently.

Not really. Watching individual neuron firings would be equivalent to watching individual parameter weights and the outputs of each step of the transformer. Token generation is literally the entire functioning of transformers.

As a communist, I expect you to understand the concept of quantity transforming into quality.

Yeah, patterns are, by definition, substrate independent. But transformers only maintain high level patterns on a per-token basis. High level patterns can and do emerge from weighted parameter space, and in many surprising ways, but they are fundamentally limited in transformers because transformers are, at base, next-token predictors so even though we get emergent high-level patterns that can, for example, sort lists, we STILL get hallucinations specifically because the high-level patterns are ephemeral on a per-token basis.

Do explain how this is different from saying that human brains are fundamentally limited in that neurons are just next state predictors.

Yes. Inference -> fitness check -> iterate. Agentic retry. It’s incredibly expensive precisely because it uses next-token predictors to generate an answer with an already-known fitness algorithm and then just re-runs inference until the answer passes the fitness test.

Except it’s not incredibly expensive because the system works on the principle of gradient dissent. It isn’t just producing a random value each turn, it produces a plausible value within the context which is precisely what allows it to quickly converge on a solution.

Yes, modern transformer harnesses do a TON of work and actually make these parrots useful instead of novelties. But it doesn’t change the fact that they are fundamentally statistically weighted parameter-space stochastic next-token predictors, no matter how much you add to them.

Exactly the way the neurons in your brain are stochastic next state predictors.

Instead of arguing against the technical reality, why not focus on the truth about the harnesses - they add a ton of value and make next-token prediction much more useful in some contexts, especially contexts like producing working code.

I would urge you to spend a bit of time actually learning about the technical reality instead of continuing to argue here.

The whole thing with R’s in strawberry hasn’t been true for a while now. Turns out you can use RL to get the model to do basic calculation. Notably, this is the exact same problem humans have. The way our brains work is also stochastic, and we struggle to do complex math in our heads. But of course, we can reinforce train ourselves to get better at it. And what we typically do is use an external aid like pen and paper to work through problems, which is basically no different from an LLM harness. If you hook up an LLM to REPL in octave, then it can do math quite well all of a sudden.

Understanding the limitations of LLMs and how to use them effectively requires moving past reductive thinking. While token generation is the base operation, focusing on that is like trying to understand the brain by looking at individual neuron firings. What’s actually interesting in both cases are the high level patterns that end up being produced which I’d argue are substrate independent. Meanwhile, a combination of an LLM with a harness can be seen as a type of a neurosymbolic system. The neural network generates novel patterns, while the symbolic engine provides the rails for it to function within.

Yeah pretty much, I’m very excited for the bubble to finally pop too.

What I’m describing is the general feedback loop. Coding is just one application here, and plenty of problems can be encoded in the same way. Again, it doesn’t actually matter if it’s getting a greater basis in reality or not. All it needs to do is to generate plausible outputs within a particular context, and these can be tested, and iterated on to solve a problem. And if you go back and read through the thread, nobody is arguing that it’s not a token generator. What’s being said is that this is a reductive way to look at what’s actually happening. It’s like saying that human is a cell reproduction machine. Technically true, and completely useless for understanding what humans do.

Or rather it produces a random output based on the input and its parameter weights

The bias is precisely what makes it not random, but rather stochastic. There’s a very big difference here.

In the form of adjustments to parameter weights

I’m talking about feedback from the environment it operates in. That’s the actual test that allows the model to keep adjusting outputs towards a specific target rather than them being random. And that’s what makes the whole thing useful in the end.

The training apparatus outside the model does this repeatedly, yes, under the thesis that tweaking parameter weights will result in fewer failures to the deterministic fitness algorithm. That’s a theory.

No, that’s not a theory, that is precisely what we measurably observe in practice with coding harnesses. And having built one myself, I can tell you for a fact that this works exactly the same way a genetic algorithm does, and large part of making an effective harness comes from ensuring that the model gets actionable feedback.

No. That’s a leap that has no basis.

The basis is me having worked on a harness and observed how the model outputs improve based on the feedback. There’s also plenty of research on the subject explaining how and why this works in detail. The parameter space is also not nearly as opaque as you seem to think.

At no point is the LLM getting closer to learning facts, and the thesis that knowledge or skill is representable as a statistical model is unproven and seems increasingly unlikely.

That’s missing the point entirely. The question isn’t about whether LLM is getting closer to learning facts. It’s about whether the biasing from the feedback loop causes the LLM to produce relevant outputs. Also, the thesis that knowledge or skill is representable as a statistical model is very much demonstrated by world models where a temporally consistent simulation of the environment is maintained.

Applying the same technique to fine tuning an LLM is a better use of time than manually fine tuning, but that doesn’t make it any less a probabilistic next-token generator that can’t represent stable facts and rules and where every fine tune for one input is always in tension with the fine tubes for all other inputs.

That’s not how any of this works at all. You’re not trying to get it to represent stable facts, you use things like compilers, test harnesses, formals specs, and so on, to create the selection pressure. Then the model is the stochastic part of the system which finds a path that satisfies the selection criteria. Or, with robotics, you have models interact with the physical world and use the feedback to adjust predictions within the model.

Yes, but just because algorithms are analogous doesn’t mean they are functionally equivalent.

Yet, they are functionally equivalent in accomplishing many tasks now. And of course, biological brains have many more subsystems and are more complex in general. I’m not arguing that part at all. My point was that what grounds our mental models in reality is the same feedback loop we use to ground LLMs, and it’s effective for the exact same reason. I also don’t think LLMs are the pinnacle of AI, they’re just one piece of the puzzle, and as I noted earlier, people are already moving towards world models now.

I find world models to be fundamentally more interesting than plain LLMs because if a model encodes the rules of how the physical world works, that provides a foundation for meaningful communication. Humans can talk to each other easily precisely because we all have a shared context which is the environment we live in. And we see how the rate of misunderstanding quickly goes up when we start talking about abstract topic because they can be interpreted in many different ways. So, if models can share the understanding of the physical world with us, it becomes a lot easier to tell them what you want, to correct them, and to have them genuinely understand requirements in a human sense.

Very well put, and people hating on this tech really need to understand that the rational thing to focus on is how to develop it outside corporate control. I wish there were more discussion regarding training models using distributed computing or optimizing local models to be more capable while using fewer resources.

The oligarchs aren’t debating AI ethics while they’re investing billions to own and control this technology, and they absolutely do not care that there’s a fringe left movement whinging about it. Our choice is to cower in nostalgia or fight to have a stake in our future. We should fight to have a say in how this technology is developed and who controls it. People who weren’t happy with proprietary operating systems didn’t just whinge about it incessantly, but rolled up their sleeves and built their own alternatives with things like Linux and BSD. That should be the guiding mindset today as well. And thanks to Chinese companies releasing open weights, we already have a good foundation to work from.

I very much agree with your argument regarding the nature of cognition and consciousness as well. People like to mystify the brain and claim there’s some intangible soul, but that’s just what we’ve always done trying to make ourselves feel special and separate from the rest of nature. The reality is that the more we learn about cognition, the more evident it becomes that it is an emergent phenomenon that exists on a gradient, from the most primitive biological feedback loops to the stunning complexity of the human mind. While LLMs are just a piece of the puzzle, they demonstrate how a lot of the things our brains are capable of can, in fact, be understood and replicated. There’s absolutely no reason to believe that we won’t be able to make systems that are capable of reasoning in the same way as us in the future.

Yes, it is a stochastic system, that’s the whole point tough. There are two things at play here, one is that the model does prediction based on the context of the current data it’s looking at rather than just whatever data it was trained on. That’s what the article is highlighting. The second part is the feedback loop such as what you see in agentic coding. The model makes a prediction, that prediction is tested against the environment, the model gets feedback, and it iterates. And that’s what grounds it in reality addressing the issue of it producing meaningless outputs. The system as a whole behaves similarly to a genetic algorithm where the solution evolves through the cycle of trial and error.

Incidentally, this is true for humans as well. This is why we need tools like the scientific method and peer review in science. People hallucinate things all the time, and when we loose the connection between the predictions the brain generates and sensory feedback we call that schizophrenia.

The key point, which is entirely correct, is that next token predictor label is technically true in a narrow sense but it completely misses what LLMs actually do. Pre training is about matching existing sequences, but RLVR means that the model generates its own sequences and learns from the outcomes instead of just imitating the training data.

The chess analogy has nothing to do with chess having finite states. What the article is saying is that a system trained only on a specific set of games would be a next move predictor, but one that chooses moves dynamically based on winning probability uses live context for its heuristic. So the mechanism encoded in that loop is able to discover and reinforce new patterns that were never part of the original training.

I grew up in USSR, and I can definitively tell you that life wasn’t based around consumption. There is a huge difference between having a decent standard of living, and having a consumerist society. Living on earth sustainably doesn’t mean we all have to revert back to primitivism.

And this all goes back to the western civilization being the problem. Its capitalist model rooted in consumerism has become the model for the whole world to emulate. And the US in particular has spent billions on building a cultural hegemony with its media glorifying American style of living. So, is it any surprise that this has become the aspirational lifestyle the rest of the world looks up to.

Had USSR prevailed, we might’ve had an alternative model to follow that’s not based on mindless consumption. But the west was hell bent on defeating USSR precisely because of the threat of an alternative. Soviets wanted to have peaceful coexistence but the west simply could not countenance that.

But there are some hopeful signs coming out of China now, for example replacing ads on billboards with pictures of scientists and descriptions of their accomplishment is a good step. https://www.globaltimes.cn/page/202609/1369560.shtml

clearly that didn’t help you much

Oh I agree, we have to move away from consumerism and growth mentality given that we live in a closed system.

🤣

Right, but trade doesn’t need to be extractive is my point. China also needs resources, but they didn’t build a system of exploitation to get them. It is possible to work with other countries without having to dominate and subjugate them so that everyone benefits.

That’s not the main reason western economies are extractive. It’s the very nature of how financial capitalism works that makes them so. https://americanaffairsjournal.org/2021/08/the-value-of-nothing-capital-versus-growth/

When I say western civilization, I’m referring to the social structures and the system as a whole rather than individuals. Of course there are plenty of people in the west who are starting to reject the system, but the west as a whole remains the same as it was institutionally. It’s extractive economic policy remains the same, and hence, as a whole it acts as a regressive force upon humanity.

And sure, China, like any human society has plenty of problems, but the difference is the overall trajectory and the social structure in China which makes it a progressive force.

I mean the whole world order the west built upon subjugation and extraction. Everything is seen as a zero sum game here, and there’s no concept of doing things collectively or finding positive sum solutions where everyone can benefit. That’s the fundamental difference from Chinese mindset, China doesn’t think that somebody has to lose for them to win. They don’t try to subjugate the rest of the world and remake it in their image. They just find ways to cooperate with other nations that are mutually beneficial. That’s how a rational society works.

And the fact that the west doesn’t see this as a collective endeavour that all humanity has to come together to accomplish shows that western civilization is a blight upon the world.

I do open source software and have been doing it for many years. I understand open source licenses very well. DeepSeek assigns all rights of the outputs to you, which makes you the legal entity with the ownership of the code:

4.2 Subject to applicable law and our Terms, you have the following rights regarding the Inputs and Outputs of the Services: (1) You retain any rights, title, and interests—if any—in the Inputs you submit; (2) We assign any rights, title, and interests—if any—in the Outputs of the Services to you.

Meanwhile, your confusion when you talk about stealing the code stems from your lack of understanding of how LLMs actually work. People tend to confuse the output looking similar to training data with the mechanism being simple retrieval. There is no database or a lookup table, and there is no copy of any text they were trained on tucked away somewhere inside the weights. What they have is a compressed statistical model of the patterns that exist across trillions of tokens the text was broken down into. When you ask an LLM a question it does not go find the relevant code and regurgitate it. What actually happens is that it generates text token by token based on the probability distribution it learned during training. It’s more akin to the way state machines like Markov chains work.

If anything, the way LLMs work is actually much closer to how humans improvise or combine ideas than to how a search engine or database works. When you write a sentence, you don’t pull up a specific piece of code you memorized from some repository and copy it. You’ve absorbed the patterns from countless examples of language over your lifetime and internalized them. Coding is a process of generating novel combinations based on those patterns.

Similarly, the weights of a model encode statistical regularities about word co occurrence, syntactic structures, semantic relationships, and higher level conceptual patterns. When it generates text it is combining those patterns in ways that were never present in the training data. That’s how LLMs produce entirely novel sentences that have never been written before, and why they can be prompted to combine ideas from different domains in creative ways.

LLMs can reproduce training data verbatim in some cases, but this happens because of overfitting on rare or repeated sequences. When a specific string appears many times in the training data like a famous quote or a common error message the model learns to assign an extremely high probability to each token in that exact sequence. So when it generates it follows the most likely path to reproduce the original text. It is similar to how a human can memorize a song or a poem by hearing it many times but that does not mean our general language ability works by pulling up stored copies from memory.

We’re talking about the worker, not the bourgeoisie. Their labour was exploited to benefit a corporate entity. And you’re consuming stolen proceeds from the bourgeoisie and defending their contributions.

You can certainly make this argument when it comes to closed corporate models. However, it completely falls apart when applied to open models that aren’t owned by any corporation. Open models are available for everyone to use, and you running Qwen on your machine isn’t benefiting any corporate entity in the slightest. And the whole stolen goods angle is basically the RIAA anti-piracy argument, you should be deeply ashamed of yourself for parroting it.

You’re also confusing tools with the actual capitalist system here which makes your entire argument deeply incoherent. It is simply not possible to avoid using anything produced by exploited labour while living in a capitalist society. And nobody is defending capitalism here.

I would urge you to spend a bit of time trying to actually understand the subject you’re attempting to discuss here to form a coherent position on it.

OpenRA is great, but that’s the older 2d C&C engine implementation. OpenTS is the newer 3d engine.