There is something to say about the power of brute force approaches. Right now scaling multi agent is seemingly what is becoming the next important axis, though very strongly constrained by quality of communication protocol and compute. Even if one model is not strong enough individually, more models and more attempts as well as differing opinions may nudge the models into a better solution basins.
Kind of an extension of the effect where you talk to a model, it is super confident (but wrong), you correct it not directly but perhaps with a question and asking it to clarify. This then in turn leads to a better approach overall.
We desperately need to keep local models competitive with the hyper scalers. I won't want to be beholden to them and their whims forever. With anthropic now restricting game reverse engineering I think we're about to cross a Rubicon of censorship.
The nice thing is that current local modes don't go away. The weights are out. Qwen 27b and deepseek are already very useful. Hardware will keep getting stronger so after the current memory crisis you can run them more cheaply.
Of course the proprietary models will continue to improve and we have no guarantee that there will be new open models. But for coding, the local models are already good enough for a vast amount of use cases, if you are an experienced developer yourself and can guide it.
This is mostly a memory bandwidth issue. Qwen 3.8 27b and the related Bonsai 2 27B run on my M5 Max laptop with 36GB of ram. It’s a little slow due to my specific model having lower memory bandwidth than higher end models with more ram and cores. MoE models perform better here and expert streaming engines can allow larger than memory models to run.
Multiple teams and efforts are now making Qwen 3.8 27b work much faster on Mac and CUDA and the inference speed is doubling almost every week.
There's now ongoing competition about who will make it work faster with exactly the same byte output at https://mlx.fast . TensorFold is the project to follow.
Its possible (though very hard to do well, and needs a fair bit of compute) to improve the local models we have, too. Continued training is possible as time goes on.
Even if Deepseek v4.1 is the last open source LLM I have access to, I would be perfectly fine with that. My secret is that I have this relatively unknown technique for solving problems where I use this quantum engine that developed from just atoms over 4.54 billion years into the most sophisticated machine in the known universe. It's a frontier model that every human has access too, actually.
So when I really want to solve a problem, I just think on it. And when I need something to iterate through the things I thought of quickly, I use deepseek.
You say that now, but you don't know what the future models will be capable of. If models get ridiculously good, you might not be perfectly fine with that.
Yep, fully satisfied with DS 4.1 Flash or GLM 5.3 Flash. I prefer using one chat at a time, actually thinking through things with it, as opposed to the YOLO run all night multi-agent thing. I mean I’m still okay with subagents if my main thread needs to explore a few options and then report back, but all on the same idea.
The thing here for me is not that I’m unable to solve problems with my brain, it’s that I value the automation in labor provided by coding agents. I do not want future advancements to be locked behind proprietary doors.
I agree with that but the cat is out of the bag now. Unless you believe in exponential improvement (which I don’t), the open models might be at a resource disadvantage, but that doesn’t mean that they can’t catch up or surpass the closed ones. Throwing money at a problem doesn’t guarantee success.
I mainly just want to point out the value of human ingenuity coupled with a model like DS4.1 is technically enough to get most people to where they want to go, and that includes smarter open models.
It can run on one strix halo machine, which is what I have. For everything else, there’s a vibrant community brute forcing more efficient engines and quantizations. There’s a minimum bar right now, so it depends on your hardware.
I bought my box for $3k last year because I hadn't updated my PC since the first threadripper 1950x, and the Strix Halo is the only x86 with unified memory that's optimized for the kind of CPU intensive stuff I do, and they finally have proper AVX512 too. It was the perfect upgrade for me (except for lack of PCIe 5 & ECC), which coincidentally happened to be perfectly positioned for local inferrence.
and I liked it so much (and saw that the RAM drought was going to last for at least a few years) so I bought a second one to use as a general server for myself and my family.
I’m all for local models and getting away from Anthropic/OAI but Qwen 27B might as well be a horse drawn carriage compared to a 2026 Porsche. Like, I guess it’s useful if you had nothing better but acting like it’s comparable to anything the frontier models have is just disingenuous.
disagree, it’s absolutely comparable. but an m5 max is where it shines. and there are 50 flags to finetune with llama server before you get it right. i do prefer llama server over mlx currently on mac as well.
I have all three and you're not going to convince me of that.
Side-by-side Qwen 3.8 27B is ~6x as fast as DS4 and GLM 5.3 and faster than that if you get the speculative engine set up properly. That means you can iterate very rapidly on the same problem and for many problems this is the way to solve them. DS4 and GLM 5.3 shine when Qwen would take too many turns to solve something, they simply have more world knowledge and seem to home in on possible solutions faster in #tg but slower in wall clock time. If a particular problem is within striking range of all three then Qwen will usually be the fastest solver by far.
It also depends on how good your prompt is, whether you are setting up points for guidance or whether you expect everything to be solved in a single shot.
Wait and see, I give it fairly high odds everything tagged as 'uncensored' or 'abliterated' will get wiped from huggingface at some point in the future. The main barrier to wider spread adoption of this right now is that very few people have the legit on-prem hardware resources/budget to run something like an abliterated Q8 of deepseek v4.1 flash 1017 or mimo 2.6 or equivalent in size locally.
Doubt it. Even if HF drops them, it will not be hard for someone to move it to something like a decentralized torrent site dedicated to open source models.
Banning them would just invite competition which is why I think HF is holding out as long as they legally can.
> I give it fairly high odds everything tagged as 'uncensored' or 'abliterated' will get wiped from huggingface at some point in the future
This is the kind of bet where I would take the opposite position without hesitation.
The only thing I could see them removing would be models that are specifically trained to do bad things, like a model trained to produce illicit underage image content. And I would be totally understanding of and encouraging of that, even if it makes the free speech absolutists unhappy.
I think the uncensored and abliterated models are more hype than actual threat. The journalists and politicians are too busy going after the AI labs every time someone does something bad and the police can find ChatGPT logs before it. I doubt we’re going to find a situation where someone was running local models and also keeping logs of everything and also commits a crime and also the journalists find it and also they make a story that catches on and so on.
It’s such a niche thing it’s just not a priority. There are some open LLM people who are treating it like a crisis before it happens, but come on. It’s the internet. We can download gigabytes of pirated TV shows despite that industry and lawmakers working hard to stop it for decades. If open models got banned somehow, you’d be able to download them the same way.
I didn't say they won't immediately move somewhere else or still be readily available. Just be taken off HF. It'll be a futile game of whack a mole for anyone trying ot stop it. The new corporate overlords at some point will face pressure. The 72GB 2160p HEVC copy of Apocalypse Now sitting on my home theatre file server which came from a well seeded torrent does indeed say that large file sizes of things that $bigcorp don't like will continue to be easily available.
Hugging Face just hosts the models. Any hypothetical censorship on their end wouldn't stop you from running the original model provided directly by the builder.
> everything tagged as 'uncensored' or 'abliterated' will get wiped from huggingface
... you do know you can download from huggingface and make your own copy, right? And even if you couldn't, you could run heretic (or whatever approach they used) yourself on the official image. It's all just a well-understood process. A decent abliteration run on DS 4.1 flash will probably cost you about $25k.
They're not small but they're hardly out of reach. I have backups on my home NAS.
> very few people have the legit on-prem hardware resources/budget to run something like an abliterated Q8 of deepseek v4.1 flash 1017
It's something like $300-400/day for the raw hardware on RunPod, a little more on AWS. Not for home users, sure, but for a business wanting to eg check its security against the best of what attackers might bring, that's nothing. And it will go down.
Yes, it'll likely just result in some people standing up a more dedicated purpose community hosting site of abliterated/uncensored open weight GGUF files. In a theoretical scenario where HF bans abliterated/uncensored models I could very well see the same people who are producing those now simply moving the location they upload stuff to.
I was able to remove much of the CCP censorship from Qwen 3.8 27B for around $90 using a LoRA. I multiplied the LoRA in permanently and listed it on HuggingFace.
The cost was approximately $80 for generating synthetic data and $10 to create and integrate the LoRA.
The main barrier to wider spread adoption of this right now is that very few people have the legit on-prem hardware resources/budget to run something like an abliterated Q8 of deepseek v4.1 flash 1017
Not as bad as you might think. This costs about as much as a decent new car nowadays.
The abliterated 1017 release I've seen is crazy good.
I guess it depends if we're talking about decent new car at $40,000 or $95,000? I've seen lots of people claiming they're running v4.1 flash 1017 on local hardware but then it turns out it's something like the unsloth Q2 XXS or similar size. The Q8 is 508GB so assuming for a moment we're not running it on a Mac, we're looking at a cost of five or six 96GB GPUs to load the whole thing with a good sized full context and KV cache.
Haven't bothered with DS4.1 personally but I've seen reports of running it with native weights on 4x DGX Sparks. Currently ~$30k but last month that setup was ~$20k.
4x 96 GB runs the NVFP4 quants, which are indistinguishable from Q8 in real-world performance for this particular model. That's not always true but DeepSeek is unusually quantization-friendly. Also remember that a big chunk of the model consists of engrams that can live in system RAM or even on an SSD if necessary.
I imagine the various GGUFs are still pretty impressive, but haven't tried them myself.
Recently saw a hand drawn cartoon to the effect of, a person is asking Claude or chatgpt to do something, it refuses, so the next step is "make the Chinese one do it", and the Chinese model is represented as a chained demon
That's pretty interesting, because with the Bonsai 2 quantisations it is quite achievable to get Qwen 27B running at 50 t/s on reasonably priced gaming machine [0]. Which in turn suggests that Fable 5 coding performance can be achieved locally with a bit of clever harnessing.
Although the same harnessing should also improve Fable's performance. The important thing here is the cap for local performance looks like it could be quite high.
> because with the Bonsai 2 quantisations it is quite achievable to get Qwen 27B running at 50 t/s on reasonably priced gaming machine
The Bonsai quantizations are not good for real work. I have tried them. They are as bad as you would expect from 2-bit quants. They may have done a good job quantizing to that limit, but it’s still bad.
Even the 4-bit Qwen3.8 27B quants are still iffy for coding work. It’s not until Q5 or Q6 that I find them useful, which is too bad because that cuts into available context on 32GB cards.
> Which in turn suggests that Fable 5 coding performance can be achieved locally with a bit of clever harnessing.
I experiment with a lot of local inference engines and I also use both Fable and Astra. I wish this was true, but it’s just not. I can generate tens of millions of Qwen3.8 tokens per day and run the hardware nonstop, but even brute forcing it like this just can’t accomplish the same thing that I can get Astra, Fable, or even Opus and Sol to get done in an hour or less most of the time. You just cannot make up for the lack of intelligence built into the model by brute force unless maybe your timeline extends to infinity.
Bonsai 2-bit quantization is notably dumber than Qwen 27B at eight bits or even four bits. I doubt you'll get this level of quality even with five of them.
Dumber yes, but not by all that much. If the multiple agent thing really improves the quality by as much as is described here, the quality loss should be nearly imperceptible.
Not sure if I agree. Remember the error rates stack up every turn of the conversation. If a 2bit model is 10% more wrong than the fp16 version, over 10 turns that adds up
Bonsai 2 has a bit different architecture as far I understand and speculative decoding is not so great yet. Before that point, original model is faster if you have the memory. Would be nice to see the speed with proper speculative decoder.
I'd wait for more benchmarks before making any conclusions since this benchmark has only "The first ten problems of their LiveCodeBench Hard list". And LiveCodeBench Hard is a several years old benchmark. Could be very saturated
I was getting deja vu. Copy paste my comment from that thread to here I guess. Apparently it's really easy to get Internet points by making up a harness and showing that good open source models can do as well as the frontier, you know, on useless saturated and contaminated benchmarks from years ago that don't reflect any of the agentic work we actually use these models for. It's so stupid because these models are at the level of a decent non fable frontier model from 3-6 months ago and it's also cool that you could probably get equal perf to gpt 5.5 from qwen 3.9 27b or gpt 5.6 sol/opus 5 from a quantized DeepSeek or glm 5.3 flash. Those models were pretty useful in their time! Instead we have whatever this new form of harness/benchmark slop is
Here the intent is to replicate GVS5H with memdoor a headless coding agent I created months ago. Anyone can reshape the harness by redefining the steps in yaml files , models and temperatures. Up to you guys to try you have the recipe in this repo.
How does it do on a more modern benchmark? Was harsh towards this in my other comment but that's a combination of this benchmark is useless, and also these open source models are pretty good and there might actually be real utility to good custom harnesses tuned for them that's getting ignored because the parity with the frontier on a saturated and contaminated benchmark makes for better headlines. I'd be curious if there's anything with these types of harnesses that make them work for realistic/relevant workflows
Remember those gpus into chips https://inference-docs.cerebras.ai/models/qwen-3.8-27b , where you have this chip that implements a model that run at those extremely high speeds... the ones that people ask "why do we need that"... if this is true... one of those chips with the correct harness is all that's needed to get fable like intelligence in every single computer in this planet...
Happy hallowen Tax Ramltman and Amowhateverwayyournameiswritten...
This is good timing. My setup at home involves a two servers with 5080s each and an Rtx spark laptop with 128gb. I plan I have qwen flash next on the latter and the 27b variant on the former. I would’ve preferred to have purchased a dgx spark but everything is just ridiculously expensive now. My own setup would be 3x the cost to buy again.
I’m mainly trying to replicate Gemini flash and muse agent for my family of 4 without breaking the bank (already had a 5080 for gaming and the other for image gen).
It is unclear if this memdoor works for agentic tasks, e.g.: “check daily if dune 3 tickets are for sale and if they are buy tickets for 4 where all 4 seats are next to each other in the same row and as center as possible.”
There is a huge cliff once you go past 128gb. Basically you need either Rtx 6000 pros (20k+) or a cluster of dgx sparks (15k+)
Benchmarks are one thing, but I've yet to see anything substantial actually built with local models. Most of the posts on X are about benchmarks and tok/s. Has anyone actually built a quality production app or game with real users?
Well, Greg KH emphatically states that local LLMs are more than good enough for bug finding in the Linux Kernel. [0] Seems strong enough for the task to me. [1]
He goes on to recommend that one uses local LLMs exclusively for bug finding because he and his colleagues have seen multiple occasions where folks who used the LLMs provided by the Major LLM Manufacturers had the stuff they were working on get "mysteriously" leaked.
Even if that sort of thing were not known to happen, it's just smart to not process data you want to keep secret in systems that are known to straight up retain user data and/or "anonymize" user data and send it back to their HQ for use in production of newer versions of their products.
[1] And even if they're six months behind the latest publicly-available tools provided by the Major LLM Manufacturers, six months ago a notable subset of folks were on here screaming that their latest tools enabled fully-automated hands-off coding and had ushered in an era of Software Factories that were rapidly going to make all programmers obsolete. ;)
I'd like to see this done with qwen 3.8-flash-next (Q8 or full precision) which beats 3.8-27B on just about every coding/command line use benchmark. In particular comparisons like terminalbench 4.0.
will definitively try it. Overall with a smarter model the improvement is in the token usage and latency. Tbh I think i could get a 10 score if i tuned the prompt and temperature a little more. The original author harness is already giving 9.6 score which is pretty good. However this harness loop can take more than 1 hour on the hardest problem at 100 token /sec. A smarter model such as 3.8-flash-next could get 10 score and resolve the hardest problem faster. That's my estimate i'll give it a try. Thx
This is somewhat tangential but speaking of ai benchmarks, I want to see one benchmarkes against passing those "filelocker" websites used for illicit downloads.
You know, the kind with 50+ ads on the page and 8 of them say download and you have to click it 5 times to get past the ads then do that on two more pages?
So someone used a small model with many requests & a massive amount of tokens, to eventually beat a large model with a single request & tiny amount of tokens. .....Cool? Were they not expecting that to work?
Fable 5 is not anymore frontier and even if, I hardly believe five very small 27b models can do like Fable 5. I think this is whishful thinking more than a fact
So far, the comments have focused on hopes for the effectiveness of open-weight models versus proprietary frontier models. That's fine, but I'm thinking about the approach and the results, as opposed to the model.
Taking the results at face value, what is it about the "team" that makes the system so effective? That's the key here: presumably, the approach should work regardless of the model, open-weight or otherwise.
My guess would be that the division of labor makes the separate contexts more effective. This is an optimization around the limits of context. Some of that is simply the net amount of context brought to bear: each team member has a context, so the net amount of context is multiplied by the number of team members. Also, since the division limits the scope of each team member's focus, the context a team member has is used more effectively.
Kind of an extension of the effect where you talk to a model, it is super confident (but wrong), you correct it not directly but perhaps with a question and asking it to clarify. This then in turn leads to a better approach overall.
Of course the proprietary models will continue to improve and we have no guarantee that there will be new open models. But for coding, the local models are already good enough for a vast amount of use cases, if you are an experienced developer yourself and can guide it.
Given pace of advance I'm not convinced this is a winning strategy. The models of today (open or not) are way smarter than those of a year ago.
...so we really do need open models to keep pace, not just be statically competitive at a given point in time
So when I really want to solve a problem, I just think on it. And when I need something to iterate through the things I thought of quickly, I use deepseek.
I mainly just want to point out the value of human ingenuity coupled with a model like DS4.1 is technically enough to get most people to where they want to go, and that includes smarter open models.
and I liked it so much (and saw that the RAM drought was going to last for at least a few years) so I bought a second one to use as a general server for myself and my family.
If expectations were static you’d be fine with qwen 2.5…but nobody sane says that today.
It’s a moving target thus we need open weights to move
DeepSeek and GLM are useful, though. Even the flash variants.
Side-by-side Qwen 3.8 27B is ~6x as fast as DS4 and GLM 5.3 and faster than that if you get the speculative engine set up properly. That means you can iterate very rapidly on the same problem and for many problems this is the way to solve them. DS4 and GLM 5.3 shine when Qwen would take too many turns to solve something, they simply have more world knowledge and seem to home in on possible solutions faster in #tg but slower in wall clock time. If a particular problem is within striking range of all three then Qwen will usually be the fastest solver by far.
It also depends on how good your prompt is, whether you are setting up points for guidance or whether you expect everything to be solved in a single shot.
https://forum.effectivealtruism.org/posts/53Gc35vDLK2u5nBxP/...
Which is literally the coolest thing to come out of this...
Did they get a letter from Nintendo's legal team or something?
Banning them would just invite competition which is why I think HF is holding out as long as they legally can.
Example: https://huggingface.co/audnai/penclaw-GLM-5.3-abliterated-fo...
Previously, I recall HuggingFace only took models down if someone complained / sent a DMCA or cease and desist.
This is the kind of bet where I would take the opposite position without hesitation.
The only thing I could see them removing would be models that are specifically trained to do bad things, like a model trained to produce illicit underage image content. And I would be totally understanding of and encouraging of that, even if it makes the free speech absolutists unhappy.
I think the uncensored and abliterated models are more hype than actual threat. The journalists and politicians are too busy going after the AI labs every time someone does something bad and the police can find ChatGPT logs before it. I doubt we’re going to find a situation where someone was running local models and also keeping logs of everything and also commits a crime and also the journalists find it and also they make a story that catches on and so on.
It’s such a niche thing it’s just not a priority. There are some open LLM people who are treating it like a crisis before it happens, but come on. It’s the internet. We can download gigabytes of pirated TV shows despite that industry and lawmakers working hard to stop it for decades. If open models got banned somehow, you’d be able to download them the same way.
... you do know you can download from huggingface and make your own copy, right? And even if you couldn't, you could run heretic (or whatever approach they used) yourself on the official image. It's all just a well-understood process. A decent abliteration run on DS 4.1 flash will probably cost you about $25k.
They're not small but they're hardly out of reach. I have backups on my home NAS.
> very few people have the legit on-prem hardware resources/budget to run something like an abliterated Q8 of deepseek v4.1 flash 1017
It's something like $300-400/day for the raw hardware on RunPod, a little more on AWS. Not for home users, sure, but for a business wanting to eg check its security against the best of what attackers might bring, that's nothing. And it will go down.
Unlikely given how trivial it is to DIY this (I think...haven't myself but from afar it seems within avg hn crowd reach)
To use an old analogy, it's not rocket science:
https://github.com/p-e-w/heretic
The cost was approximately $80 for generating synthetic data and $10 to create and integrate the LoRA.
Not as bad as you might think. This costs about as much as a decent new car nowadays.
The abliterated 1017 release I've seen is crazy good.
I imagine the various GGUFs are still pretty impressive, but haven't tried them myself.
...which is still tooo much for most people to spend on "AI hardware"....
Although the same harnessing should also improve Fable's performance. The important thing here is the cap for local performance looks like it could be quite high.
[0] https://store.steampowered.com/hwsurvey/Steam-Hardware-Softw... suggests 16GB VRAM, 32 GB system.
The Bonsai quantizations are not good for real work. I have tried them. They are as bad as you would expect from 2-bit quants. They may have done a good job quantizing to that limit, but it’s still bad.
Even the 4-bit Qwen3.8 27B quants are still iffy for coding work. It’s not until Q5 or Q6 that I find them useful, which is too bad because that cuts into available context on 32GB cards.
> Which in turn suggests that Fable 5 coding performance can be achieved locally with a bit of clever harnessing.
I experiment with a lot of local inference engines and I also use both Fable and Astra. I wish this was true, but it’s just not. I can generate tens of millions of Qwen3.8 tokens per day and run the hardware nonstop, but even brute forcing it like this just can’t accomplish the same thing that I can get Astra, Fable, or even Opus and Sol to get done in an hour or less most of the time. You just cannot make up for the lack of intelligence built into the model by brute force unless maybe your timeline extends to infinity.
5GB of bonsai weights end up being similar to 5GB of QAT weights even when one is derived from a 27B and the other from an 8B
The swarm strategy can’t overcome the dumbed down model problem. You just end up with your swarm of agents all going in the same dumbed down loops.
Happy hallowen Tax Ramltman and Amowhateverwayyournameiswritten...
Actually we tried Cerebras. It caps output at 32K tokens so the 128K reasoning (the author budget) didn't fit. Cerebras speed is real though
I’m mainly trying to replicate Gemini flash and muse agent for my family of 4 without breaking the bank (already had a 5080 for gaming and the other for image gen).
It is unclear if this memdoor works for agentic tasks, e.g.: “check daily if dune 3 tickets are for sale and if they are buy tickets for 4 where all 4 seats are next to each other in the same row and as center as possible.”
There is a huge cliff once you go past 128gb. Basically you need either Rtx 6000 pros (20k+) or a cluster of dgx sparks (15k+)
He goes on to recommend that one uses local LLMs exclusively for bug finding because he and his colleagues have seen multiple occasions where folks who used the LLMs provided by the Major LLM Manufacturers had the stuff they were working on get "mysteriously" leaked.
Even if that sort of thing were not known to happen, it's just smart to not process data you want to keep secret in systems that are known to straight up retain user data and/or "anonymize" user data and send it back to their HQ for use in production of newer versions of their products.
[0] IIRC, it is this video: <https://www.youtube.com/watch?v=NnV_cWeoo5Q>
[1] And even if they're six months behind the latest publicly-available tools provided by the Major LLM Manufacturers, six months ago a notable subset of folks were on here screaming that their latest tools enabled fully-automated hands-off coding and had ushered in an era of Software Factories that were rapidly going to make all programmers obsolete. ;)
https://artificialanalysis.ai/models/comparisons/qwen3-8-fla...
You know, the kind with 50+ ads on the page and 8 of them say download and you have to click it 5 times to get past the ads then do that on two more pages?
Taking the results at face value, what is it about the "team" that makes the system so effective? That's the key here: presumably, the approach should work regardless of the model, open-weight or otherwise.
My guess would be that the division of labor makes the separate contexts more effective. This is an optimization around the limits of context. Some of that is simply the net amount of context brought to bear: each team member has a context, so the net amount of context is multiplied by the number of team members. Also, since the division limits the scope of each team member's focus, the context a team member has is used more effectively.