On my SpacemiT K3 SBC with 32GB RAM (where models run on the eight A100 RISC-V cores with 1024 bit vectors) doing the same task I got 5, 5.8, 6.5 tok/s using gemma-4-26B-A4B-it-QAT-Q4_0.gguf, Qwen3.6-35B-A3B-Q4_K_M.gguf, Qwen3.5-35B-A3B-Q4_K_M.gguf. The corresponding dense models are more in the 2.5-3 tok/s range.
Kind of slow, but using only 14W of electricity so the Wh per task is twice as good as using my i9-13900 laptop with 4060 GPU.
I use it with a strix halo server. 35B runs stupidly fast. 27B is about 700 TPS prefill and 30 TPS token generation. Which interestedly is about what Kimi K3 gives me depending on provider.
While the laptop option is nice, for an inference server you're probably going to want the desktop form factor as it has significantly more thermal overhead and thus better performance. In the desktop models most of the internal volume is a gigantic heatsink
I have a framework desktop, but depending on your need, DGX spark might be better. The prefill and NVFP4 is a significant advantage. But framework desktop is a better general computer. I expect to be able to use it for years to come. Where as DGX Spark you’re at the mercy of NVIDIA BSP.
On an 8GB GPU and 32GB laptop: ~5 words/s while running in Qubes via ollama with completely default settings (I don't have an install at the moment that'll tell me tokens/s). Not exactly a highly tuned setup, but it's a ballpark at least :)
Tolerable and usable for some things, though thinking makes it take about a minute to reply in many cases. But getting this kind of thing to run on 8GB of VRAM is the main benefit of the mix-of-experts setup: it can do partial GPU loading and get a ton better throughput than a similarly-sized dense model (like 5-10x, sometimes more).
It's pretty fast, faster than I could type anyway, but not as fast as Claude of course. My oMLX dashboard says I get about 45 tokens per second from the Qwen model I'm running (I host it on my M1 Mac Studio, not on my laptop).
> 1. My primary concern is the risk that authoritarian governments—not solely the Chinese Communist Party (CCP), although the CCP is clearly the most capable threat—build AI models that are more powerful than those built by the US, and use them to achieve permanent military superiority or perpetrate incredibly deep repression of their own people.
What a nice implicit way to say that it would be okay if the authoritarian government is the current US government trying to perpetrate incredibly deep repression of their own people.
I have a friend who is a peace negotiator in the balkans for 20 years. He has lots of amusing (to my horrifying) stories of cat and mouse interrogations with the FSB etc as he travels between Moscow, Kyiv and the West. He uses threema and signal for most diplomatic conversations, but when he travels he only carries burner phones.
If you don't trust a government, ensure you aren't carrying any information you don't want to give up before entering their borders where you will be under their power.
based on how we've been behaving over the last 10 years, essentially yes. We are all just hoping to crawl up high enough that us and our offspring might survive while the world literally burns and billionaires and capitalism managed to destroy any movement to slow down our destruction of our environment and sustainable economies for faster short term extraction of profit and power for themselves.
Everyone pushing tools/AI during initial development/investment and building its demand in the cultural discourse always highlights its ability for good.
Now, like many tools, the majority of those selling AI to make money off of large enterprise sell its ability to increase productivity, efficiency, compliance. Either to make money or to minimise risk. And so like you say, they just become tools to make these metrics move or report them at higher granularity. And often there is either a lack of imagination or a willful ignorance of the perverse outcomes with relationship to humans because they are in service of the organisation not it's employees.
Certainly, I agree that using AI to dehumanize - generally what companies seem to be doing with it - is super bad. And it's also what is being sold to existing companies right now.
But that same AI could cause those companies to no longer exist.
The AI I'm happy about allows people without much tech knowledge create small apps to do exactly what they want. And, for those that know just a little more, use it to help them extend open source software for their niche use case.
This makes computing more personal and gives back agency to the computer operator.
Mix that with the rise of much more competition in much more custom software, and you'll see that a future can exist, if we want it, where software becomes more personal and humane.
The software vendor will capture less value, though - the margins will be thinner. Instead that value will be captured (in non-money terms) by the end users.
That also means that software companies, unable to capture so much value, must shrink and become more boutique. The software that contributes to our centralized world would lose a lot of power.
That's the future I can see. The only way it doesn't happen is if a cynical narrative wins out and manages to lock it out through regulatory capture so that only licensed operators can use or provide AI. The anti-AI narrative helps the cynics.
I agree with everything you said about their situation, but it's not like that is what will be evaluated in an IPO. There will be continued hype by the companies, lobbying to win support of a corrupt administration, and a narrative spin by clueless media about this AI revolution that will give investors fomo.
Yes their are. Majority of elected officials are just their instrument. The amount of money billionaires spend shaping media, lobbying and campaigns to get power and push propaganda to determine who is elected makes the fight to get people like AOC who are working in the interest of society so much harder.
> people like AOC who are working in the interest of society
People like AOC just pander to uneducated populace who'd prefer to use force to take free stuff of other people who work instead of working themselves.
Very easy to keep electing such politicians — just continue giving out more free stuff, so that people think it's the norm, and working is not required, just always taking is fine. People will vote accordingly
Buy up all the property in an area and raise the rents in order to force other people to give them their hard earned cash so that they can spend it on a yacht in Monaco.
They don't force anybody though. People who think that the renting is too expensive and is not worth paying, are free to either 1) move to a place that has cheaper rent, or 2) build a new house from scratch since it's cheaper than renting.
Price of rent increasing in desirable locations is not due to greed or collusion of bad actors (as long as government prevents monopolies), it's due to more people competing to live in a place that can house only a limited number. Of course everybody wants to live in big cities in the US. But rent is very cheap in dying small towns in the middle of nowhere.
>Of course everybody wants to live in big cities in the US. But rent is very cheap in dying small towns in the middle of nowhere.
Another reason why WFH should become more of the norm, it will help solve the issue of expensive housing an rent since it means people could live in these less expensive towns and still be able to work.
That’s also true of companies that AOC wants to raise taxes on. She’s not forcing anyone to pay those taxes. Companies that don’t want to pay those taxes are free to relocate away from America, abandon selling to the US market, and cut themselves off from the American labor supply. Of course every company wants to be successful in big cities in America. But taxes are very low in dying small countries in the middle of nowhere.
A lot of comments seem to not assume that ownership is equivalent to 'earning'. Why does capital need to pretend their wealth growth is equivalent to the effort of labour.
Effort is irrelevant. If it takes me ten years to grow one apple, should I get paid the same as someone who can produce 1M apples? Productivity is what matters, not effort.
People are more productive when they have tractors and seed and fertilizer and all the other requisite capital inputs for growing apples. In fact, one can hardly think of anything productive one can do with zero capital inputs.
You do not live on an island. There is trillions of dollars worth of both capital and labor at work underneath your every waking moment that supports your life. Your level of effort means nothing to anybody.
That's half my point. Because what you said might be true, and yet, the argument by Capital is often that they are working HARDER THAN EVERYONE else to get their wealth and everyone else is lazy. Which is a) both irrelevant as you state, and b) a sign they need the validation that they are putting in effort / more effort to justify what they are extracting from everyone else.
