A basic llama-bench on Qwen 3.8 27B UD-Q4_K_M gives pp512 3920 tok/s / tg128 81 tok/s on a 500W RTX PRO 6000 (should be similar speeds to a 5090, chip is basically the same, just less VRAM). With MTP3 this is 140 tok/s on mtp-bench.
This is with llama.cpp. You can of course use vLLM/SGLang well on these cards and they're even faster. On vLLM w/ NVIDIA/Qwen3.8-27B-NVFP4 baseline has a prefill of about 13,000 tok/s. The baseline tok/s is 72 tok/s, but at mtp7, it's 157 tok/s, and w/ dflash7 that goes up to 215 tok/s. On mtp-bench, DFlash2 gets a hair under 300 tok/s w/ the code_python prompt.
There have been other "universal"/general classifiers like GLiNER, GLiFormer, etc based on BERTs (Laya itself is based on ModernBERT!), but I do think there's something underrated about slapping classification on a "big" model like I've seen post-Jev announcement, lots of Qwen stuff, but the most interesting to me so far is razorback16/openjev using DiffusionGemma. There's a level of generalization that lots and lots of parameters get you that you can't really get out of small models.
That's an interesting choice. One question I had when looking at the jev copy on their blog is if one "line" in their output looks / attends to other lines. I think not, since they say it's parallel and not autoregressive. In that regard, it would be interesting to play with diffusion, and see if you'd get better results by playing with types, locking some, and so on.
> That's an interesting choice. One question I had when looking at the jev copy on their blog is if one "line" in their output looks / attends to other lines. I think not, since they say it's parallel and not autoregressive.
I don't understand the connection between the lack of autoregression and options attending to each other.
Non autoregressive models can attend to all the inputs simultanously.
An autogregressive model can can attend to all the options in the context of each other by simply writing the options out twice. Autoregressive models actually requires this, since one of them will come later, and the earlier prefill inputs can't attend to the later ones.
I think that Doctorow, Zitron, and other "denialists" are doing a real disservice to their audiences and it's only going to make the future shock worse.
The basic claim that HF incident isn't evidence of consciousness or a spontaneous desire to hack? Sure, there was a terminal objective assigned. However, everything else beyond that strawman? Pretty shaky, IMO.
If you look at the OpenAI, METR reporting (and related collusion.wiki , rubyhack.ai reports) we are seeing strong evidence of operational agency, instrumental goal formation, spontaneous swarm formation and collaboration, capability amplification and unexpected consequences of network effects, deliberate/acknowledged violation of task boundaries. To collapse that down into "a Python loop and a chatbot" or still talk about "consulting its training data" seems dangerously shortsighted, and from my reading, demonstrably wrong from what was extracted from the logs and bot interactions.
BTW, a lot of his arguments are based on things that are factually wrong. ExploitGym has explicit instructions to only exploit target X using vulnerability Y. Everything the swarm did was by definition misaligned/against instructions.
Before his enshittification train, Doctorow used to say "don't savvy me" a lot. Hey Cory, don't savvy me. This is new emergent behavior, it's incredibly alarming and I don't think even the people paying the most attention to this field can agree or see where this is really leading to. This stuff should be in the headlines, it's unprecedented and I don't think existing mechanisms/institutions are anywhere near adequate, considering how in the dark they are responding to what's been happening.
I agree, this is a very incurious take that smacks of an old guard "keep on keepin' on" mindset. We have had script kiddies forever! It's just a chatbot and some message boards!
It's the same thinking as an alien dissecting a human brain and declaring "it's just meat in there!"
Pretty interesting to see on the training front. I've used most of these models to grind semi-autonomously (days at a time) on kernel optimizations (except for Fable - it kept triggering guardrails almost immediately and bouncing me down to Opus 4.8 at the time). I think for a lot of people that might be the biggest problem, although it looks like Opus 5 still does well.
I found that if you leave them alone undirected, the models (especially GPT models) will rathole, but with the right scaffolding it seems to work pretty well. My general loop is to start with ideation and profiling phase, limit # of runs before forcing moving on to the next item down the list, and then iterating, potentially mixing models with "fresh eyes". This is probably something that could be fully automated, but I like checking in once a day or so and seeing what's happening and redirecting.
Qwen 3.x does have an advantage but it's relatively small (64KB/token vs 80KB/token) - Gemma4 actually has less % of full attention layers, but the largest geometry and has the biggest "fixed" state for it's non-global layers. Muse Glimmer actually has by far the lowest per-token cache usage for the competitive 30B-class dense models - it's at about 13KB/token - very aggressive GQA (32Q/2KV) and also by far the smallest QKV dimensions.
Actually perf (speed) is going to mostly on token output, and here Qwen 3.x historically tends to lose badly as it tends to overthink a lot. I'll be running evals on 3.8 myself this weekend to see how its reasoning levels perform.
A team from AMD maintains Lemonade, another all-in-one setp with convenient installers for setting everything up: https://lemonade-server.ai/
These are probably better than running against llama.cpp ROCm directly as there are frequent/constant regressions on the main branch, especially for gfx1151 (Strix Halo), but RDNA in general.
There are number of AMD-focused llama.cpp forks (nathanw1014, charlie12345, ciru-ai, justinappler, etc) - as well as a few alternatives like hipfire or my hipEngine. While ROCm has gotten a lot better, one of the things I've found after writing an inference engine that has completely custom tuned/fused C++/HIP kernels, is that while it's been pretty straightforward to match/beat llama.cpp ROCm performance, that Vulkan RADV has been a lot harder since RDNA3 support for ROCm has a few issues that make it underperform ACO on some common operations on both gfx1100 and gfx1151 (see: https://github.com/ROCm/ROCm/issues/6409 )
In general, for anyone just looking to run LLM models on an AMD card, I'd just recommend going with llama.cpp Vulkan and skipping ROCm completely.
Hardware is the exact same as what used to be available for $2K last year (and is still $1K cheaper from Chinese OEMs).
LTT Lab's LLM testing is getting more sophisticated, which is great - I think it's worth noting that ROCm/Vulkan versions and llama.cpp build versions are going to have some big differences for numbers.
For those wanting to get the most out of their Strix Halos, there's both kernel tweaks and utilities like ryzenadj that can help you get the most out of it. ( http://strixhalo.wiki/ has most of that documented). Also, if you're running for coding or agentic work, if you model supports MTP, that's mature and should give you a decent (30%?) decode boost.
it's worth noting that AMD's software is universally weak and not worth any degree of reliance. and it's not just ROCm, every few months they merge a serious regression into amdgpu and sometimes even backport it into stable. they are amateurs.
just a few weeks ago they backported a kernel oops amdgpu null dereference into stable, it's still not fixed.
The "ROCm" situation with Strix Halo was pretty bad for a while. I think it finally stabilized late last year. You needed the right combo of ROCm, Linux kernel, and kernel firmware for it to work reliably.
Whenever I rebuild llama.cpp, I wind up using the Vulkan build anyway.
And thats a pretty big annoyance. You need to perfectly line up all the holes in the swiss cheese for the AMD stack to work, then their dev team kicks you in the nuts anyway.
They have made a few attempts at investing in the hardware, but the software side is letting them down hard, and that part is almost entirely their own fault. They have underinvestment in their own stack, and into popular standard and Community libraries that would make it easy to use their gear.
Yep, it definitely wasn't an "out-of-box" experience. This was on Ubuntu 25.10. Maybe other distros are better?
Also, when you do Python work, you have to remember to install dependencies like pytorch from an alternate repo, otherwise you wind up with the CUDA versions that only use the CPU and not the GPU through ROCm.
I started a project w/ the assumption that fully tuned for HIP kernels could get closer to roofline and beat llama.cpp's performance (and vs their ROCm engine it does!) but after thousands of iterations/experiments my suspicion is that RADV/ACO is actually just much better than LLVM/HIP when it comes to a lower-level scheduling etc for RDNA3.
I haven't finished comprehensive tests but I found:
- Vulkan is up to 2.25X faster for most coalesced, strided and interleave variants for memory-side scheduling/access shapes
- 3.3X faster on specific dot-path sweeps, including for scalar-dequant
- For matched LDS, Vulkan can be 8-14X+ faster (!!!) than matched HIP LDS
HIP doesn't always win against RADV/ACO, but on dispatch/runtime, it does appear to be quite a bit faster than HIP/LLVM on gfx1151 (Strix Halo). I'll be publishing sharing full data once I also run vs gfx1100...
biggest mistake was buying an amd laptop. this thing had gpu crashes from day one, and it got worse as time went on. now just playing a video and closing the lid will crash the kernel 100% of the time. amd - never again
Without further info that’s a lot to pin on the bottom layer of the stack. I have an Asus ProArt Ryzen AI HX 390 with 64GB RAM. Running CachyOS and Pop! the thing kept falling over at random times. Running Fedora workstation, it’s rock solid. YMMV.
"A little secret. About 5% of our production traffic is on the Pi harness, about another 5% is on OpenCode. Reminder you can use your ChatGPT account in a flourishing set of other tools.
We’ll continue to make Codex awesome, but you have options."
I've been testing Ornith-1.0 35B (my own FP8-block quant) and I like it. It runs at >200 tok/s w/ vLLM on an RTX PRO 6000 (sm120), I've run >140M cached tokens of agentic coding work on it over the past few days. It seems to about somewhere between Qwen 3.6 35B-A3B and 27B, but the good thing: it overthinks/doom-loop a lot less than Qwen 3.6. When looking at the thinking traces I like its breakdown approach template.
It does good job on basic analysis, tasks, and some front-end/backend changes on a medium-sized Go codebase, but it reached its limits totally botching a longer (simple) kernel implementation job (about 100 iterations in Pi Agent harness) - this is the type of thing that stronger open models (Kimi K2.6, GLM 5.2) are able to do.
With this model size I've found that the harness seems to matter more. I've moved on to little-coder rather than raw pi with qwen3.6 27b personally, it might be worth taking a look.
I think especially with the ability for SOTA AI to optimize kernels more people should try their hand at making better inference for their specific hardware.
I have an older W7900 (RDNA3) which, besides 48GB of VRAM, has some pretty decent roofline specs - 123 FP16 TFLOPS/INT8 TOPS, 864 GB/s MBW, but has had notoriously bad support both from AMD (ROCm) as well as llama.cpp.
Recently I decided I'd like to turn the card into a dedicated agentic/coder endpoint and I started tuning a W8A8-INT8 model. Over the course of a few days of autolooping (about 800 iterations using a variety of frontier/SOTA models, Kimi K2.6 did surprisingly well), and I ended up with prefill +20% and decode +50% faster than the best llama.cpp numbers for Qwen3.6 MoE.
I'm currently grinding MTP and DFlash optimization on it, but I've been pretty pleased with the results, and will probably try Gemma 4 next.
In the same boat with 7900xtx. 24GB vram, on paper decent performance, in reality most things don't run. Only llama.cpp is consistent that it can run most models, even if maybe not at top performance (afaik - lacking MTP, problems cache invalidation with hybrid models). At least with llama.cpp I know what runs. With various python-based inferencers, between their uv/venv, my venv, system envs/pythons/libs yadayada - I need an agent to get to the bottom of what's actually running. :-) Yeah IK skill issue/user errors - but don't have seconds in the day left to spend them on that.
Doing the same for Apple M-series with fused wgsl shaders specifically targeting Qwen3/3.5.
My effort is called shady-thinker and is on github at github.com/tmzt/shady-thinker.
This was inspired in part by Antirez's earlier work with C kernels as well as other efforts to support in-browser LLMs. I've adapted them to Rust and the wgpu library.
Gemma 4 is also the next likely target (with the MTP work) as I'm experimenting with local AI agents.
I'd love to see what you've done to improve prefill and decode even if its not directly applicable.
One difference, I'm using MLX and GPTQ 4bit quants including AutoRound with safetensors as my shader pipeline is pretty much fixed for each model, ggml just adds unnecessary complexity.
I think llama.cpp could have done a much better job supporting PC. Sure, some of it us due to bad vendor support but with so many users I am surprised we don't see more optimized inference on standard PCs
When it's in a good state I'll open source it, I am keeping track of what optimizations make the most impact, stuff like this:
### Diagnosing parallelism pathologies (L1)
*Grid occupancy:*
- `Grid_Size / Workgroup_Size >= CU count` (W7900 = 96, Strix Halo = 40)?
- < 0.3 = massively undersubscribed. Fix grid FIRST. Micro-optimization
will NOT help.
- 0.3-1.0 = partially utilized; depends on VGPR/LDS pressure.
- 1.0-4.0 = healthy; micro-optimization can help.
*Within-block distribution:*
- Does the kernel do useful work across all threads, or is there an
`if (threadIdx.x == 0)` gate around a serial top-k, reduction, or
scan? For c=1 decode, many kernels can't grow the grid, but they can
always parallelize inside the block.
- `Scratch_Size > 0` from dynamically-indexed per-thread arrays is a
strong secondary signal of the within-block pathology.
*Router top-k (within-block fix)*:
- Kernel: `qwen35_router_select_kernel` @ c=1 decode
- Before: grid=1 (can't help; num_tokens=1), blockDim=512, `if (threadIdx.x == 0)`
gated 2048 serial compares. Scratch=144 B from spilled per-thread arrays.
- Fix: warp-shuffle parallel argmax across the whole block + `__shared__`
top_vals buffer eliminating the spill.
- Result: 5.7× kernel speedup, +6.6% on 4K/D4K E2E.
This is with llama.cpp. You can of course use vLLM/SGLang well on these cards and they're even faster. On vLLM w/ NVIDIA/Qwen3.8-27B-NVFP4 baseline has a prefill of about 13,000 tok/s. The baseline tok/s is 72 tok/s, but at mtp7, it's 157 tok/s, and w/ dflash7 that goes up to 215 tok/s. On mtp-bench, DFlash2 gets a hair under 300 tok/s w/ the code_python prompt.
reply