The local LLM hardware landscape moves fast, but every so often a combination of specialized software and consumer-adjacent hardware shifts the baseline of what is possible at home. A recent build shared on the r/LocalLLaMA community showcases eye-watering performance: 176 tokens/second decode and 6,450 tokens/second prefill using a quantized Q4 model (Qwen-3.8 Flash-Next) on a dual-GPU setup running Deadcode’s Radiance inference engine.
Achieving numbers like this requires careful orchestration of software stacks, hardware lanes, and power profiles. Let’s break down how this configuration hits these speeds and what the community discussion reveals about optimizing dual-card setups.
The Hardware and Software Stack
The configuration making waves features a custom-built rig running dual R9700 graphics cards alongside 64GB of DDR5 memory and an AMD Ryzen 9 7900X3D processor on an AM5 consumer motherboard.
Instead of standard mainstream inference runtimes, the setup leverages:
* Deadcode’s Radiance Engine: An inference engine optimized specifically for this class of hardware.
* Qwen-3.8 Flash-Next (Q4/FP8 IQ4R MoE): A highly optimized Mixture-of-Experts quantization variant hosted on Hugging Face.
* BetterBench: A throughput benchmarking tool used to capture raw performance metrics.
With aggregate throughput reaching up to 449 tokens per second across 8 concurrent requests, the setup punches well above traditional local LLM expectations.
Why PCIe Lanes and Peer-to-Peer (P2P) Matter
When running multi-GPU setups on consumer motherboards rather than enterprise Threadripper platforms, hardware bottlenecking becomes an immediate concern. Community members jumping into the thread quickly zeroed in on motherboard layouts—specifically, whether running an x8/x8 PCIe slot configuration limits performance.
According to user testing, having an x8/x8 split enables proper peer-to-peer (P2P) communication between the two cards. One user noted that moving from a single Gen5x16 / Gen4x4 layout to a properly balanced x8/x8 board (like switching between certain Asus ProArt models) avoids unnecessary penalties, ensuring the inference engine can coordinate data transfer across the cards smoothly.
Tuning, Tweaking, and Thermal Management
Raw hardware is only half the battle; squeezing out maximum tokens requires precise voltage and clock management. Community builders shared specific tuning profiles using tools like LACT to maximize efficiency:
- The Aggressive Profile: Pushing performance to its absolute limit with a 330W power limit, a -50mV undervolt, and 1349MHz VRAM yields around 195 t/s decode and 7,100 t/s prefill using alternative benchmarking tools like
llama-benchy. - The Balanced Profile: Opting for reduced fan noise and thermals by capping power at 210W with a -40mV undervolt trades away only about 7% of raw speed for a drastically quieter machine.
Naturally, high performance and community excitement have driven up market demand. Multiple commenters noted that R9700 card prices have jumped significantly—anywhere from modest regional bumps to sharp 50-60% spikes in places like Switzerland—sparking plenty of tongue-in-cheek warnings for others to stay away so individual users can scoop up a second card.
Key Takeaways
- Blazing Throughput: Utilizing Deadcode’s Radiance engine paired with a Q4 Qwen-3.8 Flash-Next model yields up to 176 t/s decode and 6,450 t/s prefill on dual R9700 cards.
- The Importance of P2P: Consumer motherboards configured with x8/x8 PCIe lanes allow for effective peer-to-peer communication, preventing bottlenecks in multi-GPU local setups.
- Finetuning via Undervolting: Tools like LACT let builders balance maximum performance (up to 195 t/s decode with aggressive clocks) against acoustic and thermal comfort.
- Market Realities: High-performing local AI hardware configurations are seeing rapid price volatility and increased demand as word spreads across communities like r/LocalLLaMA.
Original post:
https://www.reddit.com/r/LocalLLaMA/comments/1x640u6/176_ts_decode_and_6450_ts_prefill_with_q4_qwen38/
