Gpu Mods

By now you’ve probably seen the posts, the microbenchmarks, and the “RTX 4090 now hits 330 TFLOPS” screenshots. The software locks that kept consumer cards at roughly half the matrix-multiply rate of their professional counterparts on certain FP16/BF16 (and related TF32) Tensor Core paths have been…

Read MoreGpu Mods

Don’t Give Up on the Old GPU Yet: How an 8 GB RX 580 Went From Unusable to 22 Tokens a Second

The first token took almost four minutes. Then the text crawled out at 12 tokens a second, and the fan on an eight-year-old graphics card was the loudest thing in the room. Most people stop here, decide the card is too old, and either buy something newer or go back to a hosted API.

Read MoreDon’t Give Up on the Old GPU Yet: How an 8 GB RX 580 Went From Unusable to 22 Tokens a Second

A $45 GPU vs an M1 Pro, Mac Studio and RTX 5090: Local LLM Speed, Price and Power Compared

I paid $45 CAD for a used 8 GB RX 580. In an old HP Z640 workstation it runs Qwen3.6-35B-A3B, a 35-billion-parameter coding model, at 21 to 23 tokens per second on code. My 32 GB M1 Pro laptop runs the same model at 34 to 37. A Mac Studio with an M5 Max reaches about 130, and an RTX 5090 reaches…

Read MoreA $45 GPU vs an M1 Pro, Mac Studio and RTX 5090: Local LLM Speed, Price and Power Compared

The Mechanics and Limitations of Ternary Quantization in Edge Language Models

The field of local machine learning has shifted dramatically toward extreme efficiency, fueled by the demand for models that can execute entirely client-side without relying on cloud infrastructure. Innovations in model architecture and quantization (such as the release of compact frameworks like…

Read MoreThe Mechanics and Limitations of Ternary Quantization in Edge Language Models

Local Large Language Models on Low-Power ARM Hardware: Architectural Realities and Optimization Strategies

The deployment of large language models (LLMs) has historically been anchored to power-hungry x86 workstations or enterprise-grade GPU clusters. However, shifts in consumer and mobile silicon markets are changing the operational calculus. Low-power ARM-based systems-on-chip (SoCs) (traditionally…

Read MoreLocal Large Language Models on Low-Power ARM Hardware: Architectural Realities and Optimization Strategies