{"id":3641,"date":"2026-11-17T13:30:20","date_gmt":"2026-11-17T20:30:20","guid":{"rendered":"https:\/\/yyc.ninja\/?p=3641"},"modified":"2026-11-17T13:30:20","modified_gmt":"2026-11-17T20:30:20","slug":"dont-give-up-on-the-old-gpu-rx-580-8gb-tuning","status":"publish","type":"post","link":"https:\/\/yyc.ninja\/index.php\/2026\/11\/17\/dont-give-up-on-the-old-gpu-rx-580-8gb-tuning\/","title":{"rendered":"Don&#8217;t Give Up on the Old GPU Yet: How an 8 GB RX 580 Went From Unusable to 22 Tokens a Second"},"content":{"rendered":"<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1200\" height=\"630\" src=\"https:\/\/yyc.ninja\/wp-content\/uploads\/2026\/11\/cg-dont-give-up-on-the-old-gpu-rx-580-8gb-tuning-cover.png\" alt=\"Don't Give Up on the Old GPU Yet: How an 8 GB RX 580 Went From Unusable to 22 Tokens a Second\" class=\"wp-image-3640\" sizes=\"auto, (max-width: 1200px) 100vw, 1200px\" srcset=\"https:\/\/yyc.ninja\/wp-content\/uploads\/2026\/11\/cg-dont-give-up-on-the-old-gpu-rx-580-8gb-tuning-cover.png 1200w, https:\/\/yyc.ninja\/wp-content\/uploads\/2026\/11\/cg-dont-give-up-on-the-old-gpu-rx-580-8gb-tuning-cover-300x158.png 300w, https:\/\/yyc.ninja\/wp-content\/uploads\/2026\/11\/cg-dont-give-up-on-the-old-gpu-rx-580-8gb-tuning-cover-1024x538.png 1024w, https:\/\/yyc.ninja\/wp-content\/uploads\/2026\/11\/cg-dont-give-up-on-the-old-gpu-rx-580-8gb-tuning-cover-768x403.png 768w, https:\/\/yyc.ninja\/wp-content\/uploads\/2026\/11\/cg-dont-give-up-on-the-old-gpu-rx-580-8gb-tuning-cover-100x53.png 100w\" \/><\/figure>\n<h2 class=\"wp-block-heading\">Don&#8217;t Give Up on the Old GPU Yet: How an 8 GB RX 580 Went From Unusable to 22 Tokens a Second<\/h2>\n<p class=\"wp-block-paragraph\">The first token took almost four minutes. Then the text crawled out at 12 tokens a second, and the fan on an eight-year-old graphics card was the loudest thing in the room. Most people stop here, decide the card is too old, and either buy something newer or go back to a hosted API.<\/p>\n<p class=\"wp-block-paragraph\">I nearly did the same. The card in question is a Radeon RX 580 2048SP with 8 GB of VRAM, a $45 used part from 2018, sitting in an HP Z640 with two Xeon E5-2650 v4 processors and 144 GB of DDR4. It now runs a 35-billion-parameter coding model at 21 to 23 tokens a second on real code, reads prompts at 150 to 170 tokens a second, and serves a 64K context window. The first prompt of a real coding session, 24,800 tokens, went from 210 seconds to about 55.<\/p>\n<p class=\"wp-block-paragraph\">None of that came from the card. It came from finding out which of the two speeds was actually broken, and then from a short list of settings. This post is that list, in the order it matters, with the numbers behind each one and the changes that did nothing so you do not repeat them.<\/p>\n<h3 class=\"wp-block-heading\">Two numbers, and only one of them is the card<\/h3>\n<p class=\"wp-block-paragraph\">Every local inference session has two speeds, and people usually measure the wrong one.<\/p>\n<ul class=\"wp-block-list\">\n<li><strong>Decode<\/strong> is tokens written per second. It is limited by memory bandwidth, because each token means reading the active weights again. On a card with 8 GB of VRAM and a 22 GB model, most of those weights are in system memory and the CPU reads them.<\/li>\n<li><strong>Prefill<\/strong>, or prompt processing, is how fast the model reads what you sent. It is limited by compute and bandwidth together, and it is what decides time to first token.<\/li>\n<\/ul>\n<p class=\"wp-block-paragraph\">For a chat message of 200 words this hardly matters. For an agent, an attached document, or a long conversation, it is everything: an agent re-sends the whole conversation every turn, so a 25,000-token prompt is normal and the first token can take minutes.<\/p>\n<p class=\"wp-block-paragraph\">So before changing anything, read the two numbers separately. llama.cpp prints them on every request:<\/p>\n<pre><code>prompt processing, n_tokens = 2903, tg = 64.56 t\/s\n<\/code><\/pre>\n<p class=\"wp-block-paragraph\">My starting point was 26 tok\/s of prefill after a long idle, and 70 tok\/s with the card warm, against decode of 14 to 17 tok\/s. Both were bad, for different reasons. Fixing one does not fix the other, and the settings below are grouped by which one they move.<\/p>\n<p class=\"wp-block-paragraph\">One rule before the numbers. I do not rank a card or a model on published benchmark scores. Almost every figure you find online comes from a short, clean prompt, a raw API call, and a machine with nothing else running, and none of that survives contact with a real working day. What I do instead is run the same work I would be doing anyway: a coding agent on real repositories, chat sessions, document and image questions, the daily jobs my home automations send through the same endpoint. Then I rank by how it actually felt and how long it actually took. Every number in this post comes from that, including the ones that went against an online benchmark. Expect your own numbers to differ from mine and from everyone else&#8217;s, because the prompt length and the agent in front of the model decide more than the card does.<\/p>\n<h3 class=\"wp-block-heading\">The single fix that gave back 20 percent on every model<\/h3>\n<p class=\"wp-block-paragraph\">This one is worth a section of its own, because it produced the largest gain and it is invisible in every guide.<\/p>\n<p class=\"wp-block-paragraph\">An RX 580 exposes only 256 MB of its VRAM for the CPU to address directly. That small window exists to move buffers between the CPU and the card. After the GPU has been idle for a while, that window is already full when a model loads, so llama.cpp&#8217;s Vulkan backend falls back to allocating its working buffers in system memory instead, where it reaches them over PCIe at about a twentieth of the card&#8217;s own speed. The model loads, nothing errors, and every request is quietly slower until the process restarts.<\/p>\n<p class=\"wp-block-paragraph\">The backend has a switch that says never to use that window:<\/p>\n<pre><code class=\"language-ini\"># \/etc\/systemd\/system\/llama-gpu-router.service.d\/vk-vidmem.conf\n[Service]\nEnvironment=GGML_VK_DISABLE_HOST_VISIBLE_VIDMEM=1\n<\/code><\/pre>\n<p class=\"wp-block-paragraph\">With it, buffers always go into the card&#8217;s own memory. Loaded after an idle GPU, the 35B went from 17 to 21.5 tokens a second on code, four tries out of four, and nothing else changed:<\/p>\n<table>\n<thead>\n<tr>\n<th>Model<\/th>\n<th>Before<\/th>\n<th>After<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td>Qwen3.6 35B-A3B, writing code<\/td>\n<td>19.2 tok\/s<\/td>\n<td>23.8 tok\/s<\/td>\n<\/tr>\n<tr>\n<td>Qwen3.5 9B, writing<\/td>\n<td>18.2 tok\/s<\/td>\n<td>21.8 tok\/s<\/td>\n<\/tr>\n<tr>\n<td>Qwen3.5 4B, writing<\/td>\n<td>25.2 tok\/s<\/td>\n<td>31.3 tok\/s<\/td>\n<\/tr>\n<tr>\n<td>Gemma E4B, reading a 20K-token prompt<\/td>\n<td>182 tok\/s<\/td>\n<td>252 tok\/s<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<p class=\"wp-block-paragraph\">The pattern held on every model, and none of them crashed. The cost is a few hundred MB more VRAM per model.<\/p>\n<h3 class=\"wp-block-heading\">Keep the weights where the CPU can read them<\/h3>\n<p class=\"wp-block-paragraph\">The second-largest gain was keeping the model&#8217;s pages resident. My first real prompt after 11 idle hours prefilled at 26 tok\/s instead of 175, because the kernel had pushed the model&#8217;s pages out to swap and every one had to come back from the SSD before it could be read.<\/p>\n<p class=\"wp-block-paragraph\">Two settings fix that. Loading with <code>mmap+mlock<\/code> marks the model&#8217;s pages as unswappable, and the systemd unit needs <code>LimitMEMLOCK=infinity<\/code> or the kernel refuses. Then the pages never leave:<\/p>\n<ul class=\"wp-block-list\">\n<li>Decode went from 14.1 to 17.8 tokens a second.<\/li>\n<li>The first prompt after memory pressure went from 91 seconds to 20.<\/li>\n<li>Swap-in, the first request after a different model was loaded, fell to 6 to 9 seconds.<\/li>\n<\/ul>\n<p class=\"wp-block-paragraph\">The trade is that the model&#8217;s RAM is now unavailable to everything else on the machine, so size your build with that in mind. On the 35B it is about 21 GB.<\/p>\n<p class=\"wp-block-paragraph\">There is a second half to this. When the model reloads after another one, it can land on the wrong CPU&#8217;s memory. My host has two processors, and the model&#8217;s threads run on the first one, so its pages belong in that processor&#8217;s RAM. After one reload, 13.2 GB landed on the far processor and decode fell to 12 to 15 tokens a second against 19 to 21. The fix is to free page cache before loading and move any misplaced pages back:<\/p>\n<pre><code class=\"language-bash\">echo 1 &gt; \/proc\/sys\/vm\/drop_caches\nmigratepages &lt;pid&gt; &lt;target-node&gt; &lt;source-node&gt;\n<\/code><\/pre>\n<p class=\"wp-block-paragraph\">That is worth 25 to 35 percent on its own, and it is the reason to check placement after every load rather than trusting it.<\/p>\n<h3 class=\"wp-block-heading\">Batching, which is really a prefill setting<\/h3>\n<p class=\"wp-block-paragraph\">Prefill on this card started at 70 tokens a second. Raising the batch size to 2048, with <code>ubatch-size<\/code> matching it because the batch cannot be smaller than the micro-batch, took the 35B to 164:<\/p>\n<table>\n<thead>\n<tr>\n<th>Setting<\/th>\n<th>35B prefill<\/th>\n<th>4B decode (warm)<\/th>\n<th>9B decode (warm)<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td>Default, batch 512<\/td>\n<td>70 tok\/s<\/td>\n<td>22<\/td>\n<td>16.7<\/td>\n<\/tr>\n<tr>\n<td><code>batch-size 2048<\/code>, <code>ubatch-size 2048<\/code><\/td>\n<td><strong>164<\/strong><\/td>\n<td><strong>38<\/strong><\/td>\n<td><strong>23.9<\/strong><\/td>\n<\/tr>\n<tr>\n<td><code>batch-size 4096<\/code><\/td>\n<td>no gain, +1 GiB VRAM<\/td>\n<td><\/td>\n<td><\/td>\n<\/tr>\n<tr>\n<td><code>batch-size 8192<\/code><\/td>\n<td>slower, 107 tok\/s, spills to GTT<\/td>\n<td><\/td>\n<td><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<p class=\"wp-block-paragraph\">Going past 2048 is a loss on an 8 GB card because the working set no longer fits in VRAM and spills. If you change one number in your preset, change this one, and then stop.<\/p>\n<h3 class=\"wp-block-heading\">Context size without running out of VRAM<\/h3>\n<p class=\"wp-block-paragraph\">The context window is the biggest single memory decision you make, so set it on purpose rather than accepting the default. In llama.cpp it is one line in the preset:<\/p>\n<pre><code class=\"language-ini\">[*]\nc = 65536\n<\/code><\/pre>\n<p class=\"wp-block-paragraph\">That number, 65,536 tokens, is what a coding agent needs. The cache it requires grows in proportion to that token count, whether the model uses the space or not, and on an 8 GB card it is the whole reason the next paragraph exists. My own reason for 64K is measured rather than guessed: the largest real agent prompt I have seen on this stack is about 51,000 tokens, so 64K leaves room to keep working without a reload, and anything past that is memory taken from the model itself.<\/p>\n<p class=\"wp-block-paragraph\">A 64K context normally costs 2 GB or more of that cache. Quantising the key and value cache to 8 bits (<code>cache-type-k = q8_0<\/code>, the same for values) halves it, which is what makes 64,000 tokens fit on an 8 GB card at all. The cost is a small accuracy risk that did not show up in any of my tests.<\/p>\n<p class=\"wp-block-paragraph\">The trade runs the other way too, and it is the mistake I would warn you about. Larger is not safer. Every extra 32K of context is real memory that the operating system cannot give to the model&#8217;s weights or to page cache, and when the context is set to the model&#8217;s maximum of 262,144, a request that long is an out-of-memory failure after waiting for it to be accepted, not a slow answer. On one of my small models the maximum context also broke long prompts outright until the batch settings were adjusted. Set the window to the size of your actual work, check that a prompt near the limit still succeeds, and compact a long session before it reaches the ceiling rather than after.<\/p>\n<p class=\"wp-block-paragraph\">Related: the model&#8217;s thinking budget is a context hog too. At 17 tokens a second, every 1,000 tokens of reasoning costs about a minute. Setting a budget of 256 or 512 for routine work, and leaving long thinking for the few turns that need it, changes the feel of the system more than most hardware upgrades would. On the models that honour it, llama.cpp takes it from the preset as <code>reasoning-budget<\/code> or from the request as a chat template option, so a gateway alias can carry the policy.<\/p>\n<h3 class=\"wp-block-heading\">Drafting ahead, and saving the session<\/h3>\n<p class=\"wp-block-paragraph\">Two more settings that behave differently from what you might expect:<\/p>\n<ul class=\"wp-block-list\">\n<li><strong>Speculative decoding.<\/strong> The 35B ships with a small draft head (MTP). The model proposes a couple of tokens, then verifies them in one pass. Depth 2 gave a 79 percent gain. On dense models and on models without a draft head it did nothing or lost, so check whether your model has one before enabling it.<\/li>\n<li><strong>Slot save and restore.<\/strong> llama.cpp can write the prompt cache to disk and load it back. After a model swap, the next turn in a 21K-token session took 30 seconds with the restore against 180 without. On a card where a cold 25K prompt is three minutes, this is the difference between usable and not.<\/li>\n<\/ul>\n<h3 class=\"wp-block-heading\">Storage and exclusivity<\/h3>\n<p class=\"wp-block-paragraph\">Two boring items that cost people hours:<\/p>\n<ul class=\"wp-block-list\">\n<li><strong>Put the model files on an SSD.<\/strong> The same model reading from a spinning disk took 56 seconds to swap in instead of 33, and 56 seconds of silence reads as a hang.<\/li>\n<li><strong>Run one model at a time.<\/strong> A router configured with <code>--models-max 1<\/code> reloads on every switch, which costs 44 to 94 seconds on the 35B models. Pick the model for the session and stay on it. A second GPU job running alongside is worse: it steals bandwidth and can push the loaded model&#8217;s pages out.<\/li>\n<\/ul>\n<h3 class=\"wp-block-heading\">What sits in front of the model: LiteLLM aliases and the gateway fixes<\/h3>\n<p class=\"wp-block-paragraph\">The settings above make the engine fast. What a person or an agent actually talks to is a gateway, and on this stack it is doing a surprising amount of work. If your slow-inference problem is &#8220;the first message takes a minute&#8221;, some of the fix lives here rather than in llama.cpp.<\/p>\n<p class=\"wp-block-paragraph\"><strong>Aliases as policy, not just names.<\/strong> One model on the router is exposed as several aliases, each carrying its own parameters, so a client picks a behaviour by picking a name:<\/p>\n<table>\n<thead>\n<tr>\n<th>Alias<\/th>\n<th>What it sets<\/th>\n<th>Why<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td><code>qwen3.6:35b-a3b<\/code><\/td>\n<td>thinking on, 64K tokens<\/td>\n<td>&#8220;Deep&#8221;: better on some reasoning tasks, costs time<\/td>\n<\/tr>\n<tr>\n<td><code>qwen3.6:35b-a3b-fast<\/code><\/td>\n<td><code>enable_thinking: false<\/code><\/td>\n<td>The default for daily work. On the task battery this took end-to-end time from 822-885 s to 212-229 s for a 1 to 2 point accuracy loss<\/td>\n<\/tr>\n<tr>\n<td><code>qwen3.6:35b-a3b-code<\/code><\/td>\n<td>thinking off, temperature 0.2, top-p 0.95, top-k 20, repeat penalty 1.0, output cap 16,384, timeout 1,800 s<\/td>\n<td>Coding-agent steps: edits, tool calls, reading. Same 8 tasks in 138 s against 907 s<\/td>\n<\/tr>\n<tr>\n<td><code>qwen3.6:35b-a3b-code-think<\/code><\/td>\n<td>the same with a 2,048-token thinking budget<\/td>\n<td>Planning and debugging turns<\/td>\n<\/tr>\n<tr>\n<td>the other 35B, both thinking and off<\/td>\n<td>as above<\/td>\n<td>Measured 22 points higher on the coding bench<\/td>\n<\/tr>\n<tr>\n<td>LFM2.5 8B<\/td>\n<td>thinking only, no vision<\/td>\n<td>The fast model for drafts and speech<\/td>\n<\/tr>\n<tr>\n<td>the 4B, the 9B, the E4B, each with a <code>-fast<\/code> twin<\/td>\n<td>thinking off<\/td>\n<td>Small, cheap, no reload for routine calls<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<p class=\"wp-block-paragraph\">All of those aliases point at the same model file on the same router, so switching between them does not reload anything and the prompt cache is shared. That detail matters on an 8 GB card: an alias is free, a model swap costs 44 to 94 seconds.<\/p>\n<p class=\"wp-block-paragraph\"><strong>Three settings you would not guess from the alias table.<\/strong><\/p>\n<ul class=\"wp-block-list\">\n<li><strong><code>drop_params: true<\/code><\/strong> on every alias. Clients send options the local server does not understand, and without this they cause errors instead of being ignored. The temperature and repeat penalty are set per alias instead, because llama.cpp&#8217;s default temperature of 0.8 failed one of eight coding tasks while 0.2 passed all eight.<\/li>\n<li><strong><code>max_tokens<\/code> and <code>timeout<\/code> per alias.<\/strong> The output cap matters for an agent, which otherwise runs to the server&#8217;s default. The 1,800-second timeout exists for one reason: a cold 60,000-token prompt takes about 8 minutes here, and a shorter timeout kills the request after the work is done.<\/li>\n<li><strong>The exact-match fallback list is empty<\/strong> (<code>fallbacks: []<\/code>) on every local alias. If the card or the router is down, the request fails visibly instead of quietly being sent to a paid model. That is a deliberate choice, and it also means a slow local model never quietly becomes an expensive one.<\/li>\n<\/ul>\n<p class=\"wp-block-paragraph\"><strong>The gateway also fixes three things the clients get wrong.<\/strong> This is a small Python hook loaded into the gateway, and it only acts on requests carrying Open WebUI&#8217;s tool fingerprint, so API callers are untouched:<\/p>\n<ol class=\"wp-block-list\">\n<li><strong>Open WebUI attaches 26 built-in tool schemas to every chat<\/strong>, about 5,066 tokens. On a card that prefills at 150 tokens a second, that is 34 seconds of work before the first useful token, every message. The hook removes them and inserts one web search tool instead.<\/li>\n<li><strong>Web search works around a streaming bug.<\/strong> LiteLLM&#8217;s own search interception only runs its loop for non-streaming requests, and chat clients always stream. The hook runs the search loop itself: the model asks for a search, the hook queries a self-hosted SearXNG instance, feeds the results back, and keeps streaming the answer to the user. The search tool call is hidden from the transcript, so the client sees an ordinary answer with citations, not a tool exchange.<\/li>\n<li><strong>Background tasks are moved off the slow model.<\/strong> Open WebUI generates a chat title, tags and follow-up questions after every answer, and it was doing that with thinking on, holding the single GPU slot for 70 seconds after each reply. The hook turns thinking off for those calls and hands them to a 1.7B model on the CPU router, so the card is released to the user immediately.<\/li>\n<\/ol>\n<p class=\"wp-block-paragraph\"><strong>What the search costs.<\/strong> Prompt-processing time is the budget, and the tool schema for one search function is a fraction of the 26 it replaces. It also means no outbound call to a search provider from the card itself: the search runs against a computer on the same network.<\/p>\n<h3 class=\"wp-block-heading\">The part that surprised me most: the model matters more than the card<\/h3>\n<p class=\"wp-block-paragraph\">After all of the above, I ran five models through the same work I actually do: a real coding agent driven through opencode, pointed at five small repositories with real bugs, graded by hidden tests, with a 30-minute limit per attempt and two rounds each. Not a raw API call, not a published score. The agent sends 25,000-token prompts, reads files, runs the test suite, and fails the way it fails on a working day, which is the only way I trust a ranking.<\/p>\n<table>\n<thead>\n<tr>\n<th>Model<\/th>\n<th>Decode<\/th>\n<th>Checks passed, thinking off<\/th>\n<th>Checks passed, thinking on<\/th>\n<th>Attempts passing every check<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td>Ornith-1.5 35B, mixture of experts<\/td>\n<td>18 tok\/s<\/td>\n<td>67 of 82<\/td>\n<td><strong>80 of 82<\/strong><\/td>\n<td>8 of 10<\/td>\n<\/tr>\n<tr>\n<td>Qwen3.6 35B, mixture of experts<\/td>\n<td>18 tok\/s<\/td>\n<td>58 of 82<\/td>\n<td>52 of 82<\/td>\n<td>4 of 10 at best<\/td>\n<\/tr>\n<tr>\n<td>Qwen3.5 9B, dense<\/td>\n<td>22 tok\/s<\/td>\n<td>24 of 82<\/td>\n<td>26 of 82<\/td>\n<td>0 of 10<\/td>\n<\/tr>\n<tr>\n<td>Gemma E4B<\/td>\n<td>25 tok\/s<\/td>\n<td>14 of 66<\/td>\n<td>18 of 74<\/td>\n<td>0 of 10<\/td>\n<\/tr>\n<tr>\n<td>LFM2.5 8B<\/td>\n<td><strong>76 tok\/s<\/strong><\/td>\n<td>10 of 74<\/td>\n<td>always thinks<\/td>\n<td>0 of 10<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<p class=\"wp-block-paragraph\">Switching to the other 35B, at exactly the same speed, on the same card, moved coding accuracy from 52 to 80 checks out of 82. That is more than every hardware change in this post combined, and it cost nothing but a download.<\/p>\n<p class=\"wp-block-paragraph\">The fastest model is the worst coder. LFM2.5 runs at 76 tokens a second, four times the 35B, loads in 12 to 33 seconds and fits entirely in 8 GB. It also scored 10 of 74. It is ideal for quick drafts and speech to text. It is not an agent.<\/p>\n<p class=\"wp-block-paragraph\">Why mixtures of experts work on a small card: a 35B model with 3 billion active parameters reads only about 1.7 GB per token instead of the 22 GB a dense model of the same size would move. The rest of the experts sit in system memory and are read as needed. That is the single design decision that makes a large model viable on old hardware.<\/p>\n<p class=\"wp-block-paragraph\">Quantisation follows the same logic, with a twist. Going from 4-bit to 3-bit saved 0.6 GiB of VRAM and 20 percent of decode speed, because the CPU does more work per byte for the newer formats. The smaller file is not the faster file when the processor is the bottleneck. Stay at 4-bit.<\/p>\n<h3 class=\"wp-block-heading\">The hardware fixes people recommend, and what they did here<\/h3>\n<p class=\"wp-block-paragraph\">Three suggestions come up in every thread about slow local inference. All three were tried on this machine.<\/p>\n<p class=\"wp-block-paragraph\"><strong>Resizable BAR.<\/strong> It lets the CPU see all of the card&#8217;s VRAM instead of the 256 MB slice. It does not make the link faster, and it would have fixed the buffer problem above, but only by accident. This Z640 cannot do it: it gives the card 384 MB of address space below the 4 GB mark, and an 8 GB window cannot fit there. Enabling 64-bit MMIO in the stock BIOS opened a 2 GB window above 4 GB, the card moved into it, and it still received only 256 MB, because the driver asks for all 8 GB or nothing. The remaining option is a community-modified BIOS with no success report for this card and one report of a machine stuck in a power cycle. I skipped it, and one environment variable did the same job.<\/p>\n<p class=\"wp-block-paragraph\"><strong>Filling memory channels in a particular order.<\/strong> The advice is to use identical sticks, and my board had one 32 GB stick beside three 16 GB ones, which created a slow section of memory above the 64 GB mark. I swapped it so all four channels match, and measured again: no change at all, 21 tokens a second before and after. The slow section had held about 11 percent of the model, and the model was not waiting on memory bandwidth to begin with. For a new build, fill every channel, because an empty channel is a large loss. A mismatched stick, once the channels are all populated, is not.<\/p>\n<p class=\"wp-block-paragraph\"><strong>PCIe performance mode and other BIOS switches.<\/strong> Enabled, no measurable effect, because the link was already running at full speed with power management disabled.<\/p>\n<h3 class=\"wp-block-heading\">What to try when a setting makes it worse<\/h3>\n<p class=\"wp-block-paragraph\">For completeness, because a list of wins teaches less than a list of losses:<\/p>\n<ul class=\"wp-block-list\">\n<li><strong><code>moe-cache-mib<\/code><\/strong> (keep recently used experts in VRAM): three to four times slower at 1 and 2 GB. The upload cost per miss is higher than the reuse saves.<\/li>\n<li><strong><code>spec-draft-p-min<\/code><\/strong> above zero: raises draft acceptance as designed and still loses, because drafting less costs more than bad drafts do.<\/li>\n<li><strong>3-bit quantisation:<\/strong> 20 percent slower, covered above.<\/li>\n<li><strong>Batch sizes of 4096 and 8192:<\/strong> nothing gained, and 8192 spilled to system memory and dropped to 107 tokens a second.<\/li>\n<li><strong>A warm-up GPU job before loading<\/strong>, to dodge the idle bug: worked two times out of three. The environment variable fixed it properly.<\/li>\n<\/ul>\n<h3 class=\"wp-block-heading\">What it feels like now, and what it will not do<\/h3>\n<p class=\"wp-block-paragraph\">A cold 24,800-token coding prompt takes about 55 seconds instead of 210. Later turns take 17 to 27 seconds, and restoring a saved prompt after a model swap takes 30 instead of 180. Code generation runs at 21 to 23 tokens a second, prose at 17 to 18.<\/p>\n<p class=\"wp-block-paragraph\">For scale, a 32 GB M1 Pro laptop runs the same model at 34 to 37 tokens a second, a Mac Studio with an M5 Max at about 130, and an RTX 5090 at 183 to 271. So a tuned RX 580 reaches about 60 percent of a five-year-old laptop and about a sixth of a modern desktop card. It does not reach either. What it does is run the model at all, on a card that costs less than a licensed copy of most software, inside a machine that also serves as a home server.<\/p>\n<p class=\"wp-block-paragraph\">Two limits stay. A dense model larger than about 8 GB will not run, because it cannot be split. And prompt reading stays 15 to 70 times slower than a 5090, which is the gap that a coding agent makes you feel.<\/p>\n<p class=\"wp-block-paragraph\">The power cost is the other honest number. During a real coding session my machine drew 235 W at the wall-side sensors, about 10 to 11 joules per token, against roughly 1.4 on the M1 Pro. If you run this all day, that difference is a real bill.<\/p>\n<h3 class=\"wp-block-heading\">The order to do this in<\/h3>\n<ol class=\"wp-block-list\">\n<li>Measure decode and prefill separately, before changing anything.<\/li>\n<li>Set <code>GGML_VK_DISABLE_HOST_VISIBLE_VIDMEM=1<\/code>. It costs nothing and gave 20 percent here.<\/li>\n<li>Set <code>batch-size<\/code> and <code>ubatch-size<\/code> to 2048. Expect prefill to more than double.<\/li>\n<li>Load with <code>mmap+mlock<\/code>, and raise <code>LimitMEMLOCK<\/code> for the service.<\/li>\n<li>Set the context window (<code>c<\/code>) to the size of your real work, not the model&#8217;s maximum, and quantise the KV cache to <code>q8_0<\/code> so that window fits.<\/li>\n<li>Enable speculative decoding if your model ships a draft head.<\/li>\n<li>Put the model files on an SSD, and run one model at a time.<\/li>\n<li>Check which CPU&#8217;s memory the model landed in, and move it back if it is wrong.<\/li>\n<li>Then, and only then, change the model. Try a mixture of experts at 4-bit before you spend on hardware.<\/li>\n<\/ol>\n<p class=\"wp-block-paragraph\">The card was never the problem. The problem was that &#8220;it is slow&#8221; hid two different speeds, one of which was being halved by a buffer allocation nobody logs, and the other of which was being set by a model choice that any download could change.<\/p>\n<h3 class=\"wp-block-heading\">Key takeaways<\/h3>\n<ul class=\"wp-block-list\">\n<li>Published benchmarks are a starting point, not a ranking. Test with your own agent on your own work and rank by how it feels and how long it takes.<\/li>\n<li>Two speeds matter and they are fixed differently: writing (decode) and reading the prompt (prefill). Measure both before changing anything.<\/li>\n<li>The largest single gain was an environment variable, <code>GGML_VK_DISABLE_HOST_VISIBLE_VIDMEM=1<\/code>, worth 20 to 24 percent on every model, caused by a 256 MB video memory window filling up after the card sat idle.<\/li>\n<li>Locking the weights in memory removed the first-prompt penalty after memory pressure (91 s to 20 s), and checking which CPU&#8217;s memory the model landed in is worth another 25 to 35 percent after a reload.<\/li>\n<li>Batch 2048 took prompt reading from 70 to 164 tokens a second. The context window (<code>c<\/code>) should be sized to your real prompts, not the model&#8217;s maximum, and an 8-bit cache is what lets 64K tokens fit in 8 GB. 4-bit quantisation beats 3-bit here by 20 percent of decode speed.<\/li>\n<li>A gateway in front of the model earns its place: per-alias settings for thinking and sampling, an output cap and a long timeout for agents, and no paid fallback when the local model is down.<\/li>\n<li>The model choice moved coding accuracy from 52 to 80 of 82 checks at the same speed. The fastest model tested, four times quicker, scored 10 of 74.<\/li>\n<\/ul>\n<h3 class=\"wp-block-heading\">The concepts in plain language<\/h3>\n<ul class=\"wp-block-list\">\n<li><strong>Decode and prefill:<\/strong> writing speed versus reading speed. People quote the first and feel the second.<\/li>\n<li><strong>VRAM and system memory:<\/strong> the card&#8217;s own fast memory versus the machine&#8217;s RAM. When a model is larger than VRAM, the excess is read by the CPU on every token, at a fraction of the speed.<\/li>\n<li><strong>Mixture of experts:<\/strong> a large model where each token uses only a small part of its weights, so less has to be read per token.<\/li>\n<li><strong>Quantisation:<\/strong> storing weights in fewer bits. Smaller files and less VRAM, and not always faster, because the CPU works harder per byte.<\/li>\n<li><strong>Speculative decoding:<\/strong> the model guesses the next few tokens cheaply, then checks them all at once.<\/li>\n<li><strong><code>mlock<\/code>:<\/strong> a request to the operating system to keep a copy of memory out of swap.<\/li>\n<li><strong>NUMA:<\/strong> a machine with more than one processor, where each processor has its own RAM and reaching the other processor&#8217;s RAM is slower.<\/li>\n<li><strong>Gateway and alias:<\/strong> a proxy in front of the model that gives one model several named entry points, each with its own settings, so clients get a behaviour without knowing the details.<\/li>\n<li><strong>Fallback list:<\/strong> what a gateway does when the first model fails. Left empty, the request fails instead of being sent to a paid model.<\/li>\n<\/ul>\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h2 class=\"wp-block-heading\">Get the next one in your inbox<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Enjoyed this? I write about local AI, self-hosting, and cloud cost ninjutsu. Join the Ninja Dispatch for occasional, high-signal notes. No fluff.<\/p>\n\n\n\n\t\t\t<form id=\"sib_signup_form_2\" method=\"post\" class=\"sib_signup_form\">\n\t\t\t\t<div class=\"sib_loader\" style=\"display:none;\"><img\n\t\t\t\t\t\t\tsrc=\"https:\/\/yyc.ninja\/wp-includes\/images\/spinner.gif\" alt=\"loader\"><\/div>\n\t\t\t\t<input type=\"hidden\" name=\"sib_form_action\" value=\"subscribe_form_submit\">\n\t\t\t\t<input type=\"hidden\" name=\"sib_form_id\" value=\"2\">\n                <input type=\"hidden\" name=\"sib_form_alert_notice\" value=\"Please fill out all required fields.\">\n                <input type=\"hidden\" name=\"sib_form_invalid_email_notice\" value=\"Please enter a valid email address.\">\n                <input type=\"hidden\" name=\"sib_security\" value=\"ec07c95cd0\">\n\t\t\t\t<div class=\"sib_signup_box_inside_2\">\n\t\t\t\t\t<div style=\"\/*display:none*\/\" class=\"sib_msg_disp\">\n\t\t\t\t\t<\/div>\n                    \t\t\t\t\t<p class=\"sib-email-area\">\n    <input type=\"email\" class=\"sib-email-area\" name=\"email\" placeholder=\"you@example.com\" required=\"required\">\n<\/p>\n<p class=\"sib-FIRSTNAME-area\">\n    <input type=\"text\" class=\"sib-FIRSTNAME-area\" name=\"FIRSTNAME\" placeholder=\"First name (optional)\">\n<\/p>\n<input type=\"hidden\" class=\"sib-SOURCE_DOMAIN-area\" name=\"SOURCE_DOMAIN\" value=\"yyc.ninja\">\n<input type=\"hidden\" class=\"sib-SIGNUP_SOURCE-area\" name=\"SIGNUP_SOURCE\" value=\"post\">\n<p>\n    <input type=\"submit\" class=\"sib-default-btn\" value=\"Join the Ninja Dispatch\">\n<\/p>\n<p class=\"ninja-fineprint\">No spam. Unsubscribe anytime.<\/p>\t\t\t\t<\/div>\n\t\t\t<\/form>\n\t\t\t<style>\n\t\t\t\tform#sib_signup_form_2{background:#192a3d;border-radius:10px;padding:16px;font-family:inherit}form#sib_signup_form_2 input[type=text],form#sib_signup_form_2 input[type=email],form#sib_signup_form_2 select{width:100%;box-sizing:border-box;margin:0 0 10px 0;padding:10px 12px;border:1px solid #33465c;border-radius:8px;background:#0f1c29;color:#e8eef5;font-size:14px}form#sib_signup_form_2 input::placeholder{color:#8a9bb0}form#sib_signup_form_2 .sib-default-btn{width:100%;margin:2px 0 0 0;padding:11px 14px;background:#e63946;color:#ffffff;border:0;border-radius:8px;font-size:15px;font-weight:600;cursor:pointer}form#sib_signup_form_2 .sib-default-btn:hover{background:#d02f3c}form#sib_signup_form_2 p{margin:0 0 10px 0}form#sib_signup_form_2 .ninja-fineprint{margin:8px 0 0;font-size:11px;color:#8a9bb0;text-align:center}form#sib_signup_form_2 p.sib-alert-message {\n    padding: 6px 12px;\n    margin-bottom: 20px;\n    border: 1px solid transparent;\n    border-radius: 4px;\n    -webkit-box-sizing: border-box;\n    -moz-box-sizing: border-box;\n    box-sizing: border-box;\n}\nform#sib_signup_form_2 p.sib-alert-message-error {\n    background-color: #f2dede;\n    border-color: #ebccd1;\n    color: #a94442;\n}\nform#sib_signup_form_2 p.sib-alert-message-success {\n    background-color: #dff0d8;\n    border-color: #d6e9c6;\n    color: #3c763d;\n}\nform#sib_signup_form_2 p.sib-alert-message-warning {\n    background-color: #fcf8e3;\n    border-color: #faebcc;\n    color: #8a6d3b;\n}\n\t\t\t<\/style>\n\t\t\t\n","protected":false},"excerpt":{"rendered":"<p>The first token took almost four minutes. Then the text crawled out at 12 tokens a second, and the fan on an eight-year-old graphics card was the loudest thing in the room. Most people stop here, decide the card is too old, and either buy something newer or go back to a hosted API.<\/p>\n","protected":false},"author":1,"featured_media":3640,"comment_status":"","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[12],"tags":[],"class_list":["post-3641","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-how-to"],"featured_image_urls_v2":{"full":["https:\/\/yyc.ninja\/wp-content\/uploads\/2026\/11\/cg-dont-give-up-on-the-old-gpu-rx-580-8gb-tuning-cover.png",1200,630,false],"thumbnail":["https:\/\/yyc.ninja\/wp-content\/uploads\/2026\/11\/cg-dont-give-up-on-the-old-gpu-rx-580-8gb-tuning-cover-150x150.png",150,150,true],"medium":["https:\/\/yyc.ninja\/wp-content\/uploads\/2026\/11\/cg-dont-give-up-on-the-old-gpu-rx-580-8gb-tuning-cover-300x158.png",300,158,true],"medium_large":["https:\/\/yyc.ninja\/wp-content\/uploads\/2026\/11\/cg-dont-give-up-on-the-old-gpu-rx-580-8gb-tuning-cover-768x403.png",768,403,true],"large":["https:\/\/yyc.ninja\/wp-content\/uploads\/2026\/11\/cg-dont-give-up-on-the-old-gpu-rx-580-8gb-tuning-cover-1024x538.png",1024,538,true],"RoboGalleryMansoryImagesCenter":["https:\/\/yyc.ninja\/wp-content\/uploads\/2026\/11\/cg-dont-give-up-on-the-old-gpu-rx-580-8gb-tuning-cover-600x630.png",600,630,true],"RoboGalleryPreload":["https:\/\/yyc.ninja\/wp-content\/uploads\/2026\/11\/cg-dont-give-up-on-the-old-gpu-rx-580-8gb-tuning-cover-100x53.png",100,53,true],"1536x1536":["https:\/\/yyc.ninja\/wp-content\/uploads\/2026\/11\/cg-dont-give-up-on-the-old-gpu-rx-580-8gb-tuning-cover.png",1200,630,false],"2048x2048":["https:\/\/yyc.ninja\/wp-content\/uploads\/2026\/11\/cg-dont-give-up-on-the-old-gpu-rx-580-8gb-tuning-cover.png",1200,630,false]},"post_excerpt_stackable_v2":"<p>The first token took almost four minutes. Then the text crawled out at 12 tokens a second, and the fan on an eight-year-old graphics card was the loudest thing in the room. Most people stop here, decide the card is too old, and either buy something newer or go back to a hosted API.<\/p>\n","category_list_v2":"<a href=\"https:\/\/yyc.ninja\/index.php\/category\/how-to\/\" rel=\"category tag\">How To Ninja<\/a>","author_info_v2":{"name":"YYC Ninja","url":"https:\/\/yyc.ninja\/index.php\/author\/admin\/"},"comments_num_v2":"0 comments","_links":{"self":[{"href":"https:\/\/yyc.ninja\/index.php\/wp-json\/wp\/v2\/posts\/3641","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/yyc.ninja\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/yyc.ninja\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/yyc.ninja\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/yyc.ninja\/index.php\/wp-json\/wp\/v2\/comments?post=3641"}],"version-history":[{"count":1,"href":"https:\/\/yyc.ninja\/index.php\/wp-json\/wp\/v2\/posts\/3641\/revisions"}],"predecessor-version":[{"id":3649,"href":"https:\/\/yyc.ninja\/index.php\/wp-json\/wp\/v2\/posts\/3641\/revisions\/3649"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/yyc.ninja\/index.php\/wp-json\/wp\/v2\/media\/3640"}],"wp:attachment":[{"href":"https:\/\/yyc.ninja\/index.php\/wp-json\/wp\/v2\/media?parent=3641"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/yyc.ninja\/index.php\/wp-json\/wp\/v2\/categories?post=3641"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/yyc.ninja\/index.php\/wp-json\/wp\/v2\/tags?post=3641"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}