{"id":3407,"date":"2025-09-02T12:23:00","date_gmt":"2025-09-02T18:23:00","guid":{"rendered":"https:\/\/yyc.ninja\/index.php\/2025\/09\/02\/building-local-ai-that-actually-works-the-architecture-of-reliability\/"},"modified":"2026-10-14T23:47:15","modified_gmt":"2026-10-15T05:47:15","slug":"building-local-ai-that-actually-works-the-architecture-of-reliability","status":"publish","type":"post","link":"https:\/\/yyc.ninja\/index.php\/2025\/09\/02\/building-local-ai-that-actually-works-the-architecture-of-reliability\/","title":{"rendered":"Building Local AI That Actually Works: The Architecture of Reliability"},"content":{"rendered":"
\"Building<\/figure>\n

A blueprint for executive and engineering teams to move from local AI prototypes to production-grade, scalable, and trustworthy systems.<\/strong><\/p>\n

\n

The Illusion of Competence<\/h3>\n

The pitch for local AI is irresistible: absolute privacy, zero API costs, and sub-second latency. You download a 30-billion-parameter model, fire up a sleek UI, and ask it to write a Python script or summarize a legal contract. The result is fluent, confident, and grammatically perfect.<\/p>\n

Then you use it in production.<\/p>\n

The model forgets what it was doing halfway through a task. It invents code APIs that don’t exist. It confidently cites case law that never happened. It botches tool calls, crashes your memory, and forces you right back to paying OpenAI or Anthropic.<\/p>\n

Here is the hard truth:<\/strong> The model is not the problem. The scaffolding is. <\/p>\n

Local AI fails because teams treat it like a software API rather than a system. Running a frontier model locally is an exercise in extreme systems engineering. You are fighting unified memory constraints, KV cache bombs, context drift, and the model’s innate desire to hallucinate when it doesn’t know something.<\/p>\n

This post synthesizes the best practices for building local AI\u2014spanning coding agents, general enterprise tasks, and high-stakes legal tech\u2014into a unified architecture. We will cover the hardware, the inference stack, the guardrails, and the scaling strategies required to make local AI actually work.<\/p>\n

\n

Phase 1: The Hardware Reality \u2014 Memory is Everything<\/h3>\n

The biggest mistake teams make is treating Unified Memory (Apple Silicon) or VRAM (AMD\/NVIDIA) as an infinite pool. It is not. It is a zero-sum battlefield where model weights, context caches, and application overhead fight to the death.<\/p>\n

The Problem: The GPU War<\/h4>\n

If you run a 14B model (8GB weights) and give it a 64,000-token context window, the KV cache (the model’s “working memory” for that conversation) can easily consume another 8GB. Now add your OS, your browser, and a voice transcription engine. The system runs out of memory, forcefully evicts the model, and your next chat takes 45 seconds to cold-start.<\/p>\n

The Best Practice: Honest Budgeting<\/h4>\n

You must set hard memory ceilings.
\n* Apple Silicon (32GB):<\/strong> Cap your inference engine at 18GB. Pin your daily-driver model (e.g., a 14B Coder at 7.7GB) and a tiny utility model (1.7B at 1GB). Leave the remaining 10GB for KV cache, the OS, and transient loads.
\n* The Context Window Trade-off:<\/strong> A 131,000-token context window sounds great, but pre-allocating memory for it will eat 15GB of VRAM before you even type a word. Cap context windows to 32K or 64K.<\/strong> Use RAG (Retrieval-Augmented Generation) to fetch relevant text, rather than stuffing the whole company wiki into the prompt.<\/p>\n

The Decision:<\/strong> Limit max_context_window<\/code> per model role. A “Coder” gets 32K. An “Architect” gets 64K. This simple config change drops prefill time from 15 seconds to 2 seconds and prevents OOM crashes.<\/p>\n

\n

Phase 2: The Multi-Tasking Dilemma \u2014 One Engine, Many Hats<\/h3>\n

Teams quickly realize that coding tasks require rigid, deterministic AI, while brainstorming tasks require creative, stochastic AI. The instinct is to load two models. On 32GB or 48GB hardware, this is a fast path to OOM.<\/p>\n

The Best Practice: The “Personality Dial” via LiteLLM<\/h4>\n

You don’t need two models; you need dynamic sampling profiles. <\/p>\n

Install LiteLLM<\/strong> as a proxy in front of your inference engine (vLLM or Ollama). LiteLLM lets you define “Virtual Models” (roles) that route to the exact same physical model but inject different parameters.<\/p>\n

    \n
  • role\/coder<\/code>:<\/strong> Routes to Model-14B<\/code>. Forces temperature: 0.1<\/code>, repetition_penalty: 1.15<\/code>. Strict, deterministic.<\/li>\n
  • role\/creative<\/code>:<\/strong> Routes to Model-14B<\/code>. Forces temperature: 0.85<\/code>, presence_penalty: 0.3<\/code>. Fluent, diverse.<\/li>\n<\/ul>\n

    The Decision:<\/strong> One pinned model, multiple personalities. This costs zero extra VRAM and allows your local box to serve both developers and marketing teams simultaneously.<\/p>\n

    \n

    Phase 3: Defeating Hallucinations \u2014 Mechanical Guardrails<\/h3>\n

    LLMs are confabulation engines. When they don’t know a fact, they guess. In coding, they invent GitHub URLs and CLI commands. In law, they invent case citations. Text-only instructions (“Do not hallucinate”) fail because the model cannot distinguish between what it knows and what it guesses.<\/p>\n

    The Coding Fix: Grammar Constraints<\/h4>\n

    Instead of asking the model nicely to output valid JSON or a valid Git diff, force it to.<\/strong> Modern inference engines support GBNF grammars (via xgrammar<\/code>). You define a mathematical rule for the output format. The sampler physically cannot select a token that violates the grammar.<\/p>\n

      \n
    • Result:<\/strong> Malformed code diffs drop from 15% to 0%. Your validation loops stop chasing syntax errors.<\/li>\n<\/ul>\n

      The Legal Fix: Citation Verification<\/h4>\n

      In high-stakes domains, you must assume the model will fabricate citations. Your post-processing pipeline must mechanically verify them.
      \n1. Prompt for Tags:<\/strong> Train or prompt the model to output citations as [CITE: Smith v. Jones, 2024]<\/code>.
      \n2. Regex Extraction:<\/strong> Use a script to strip everything inside the [CITE: ...]<\/code> tags.
      \n3. RAG Cross-Check:<\/strong> Verify every extracted citation against the actual documents provided in the RAG context.
      \n4. Enforcement:<\/strong> If the citation is not in the source documents, replace it with [CITATION NOT FOUND IN SOURCE]<\/code>.<\/p>\n

      The Decision:<\/strong> Never trust the model’s obedience. Trust mechanical verification. Use Linters for code, Eyecite + Python for law.<\/p>\n

      \n

      Phase 4: Context Drift \u2014 The Goldfish Memory<\/h3>\n

      When an agent works on a 20-step task, it fills its context window. By step 15, it has forgotten the original goal. It starts refactoring unrelated code or contradicting its own plan. This is “Context Drift.”<\/p>\n

      The Best Practice: Externalize State (The State Machine)<\/h4>\n

      Do not keep task state in the context window. Write it to disk.<\/p>\n

      Implement a PLAN \u2192 PATCH \u2192 TEST \u2192 REVIEW<\/strong> loop driven by a STATE.json<\/code> file on the filesystem.<\/p>\n

        \n
      1. PLAN (Architect Model):<\/strong> Reads the goal, writes a plan to .plan\/task.md<\/code>, updates STATE.json<\/code> to phase: \"patch\"<\/code>.<\/li>\n
      2. PATCH (Coder Model):<\/strong> Reads only<\/em> the plan and the specific files listed in STATE.json<\/code>. Makes the edit (max 100 lines). Updates STATE.json<\/code> to phase: \"test\"<\/code>.<\/li>\n
      3. TEST (Shell\/Linter):<\/strong> Runs mechanically. Updates STATE.json<\/code> with pass<\/code> or fail<\/code>.<\/li>\n
      4. REVIEW (Fresh-Context LLM):<\/strong> A separate LLM with no prior context<\/em> reviews the diff and test result. Accepts or sends it back to PATCH.<\/li>\n<\/ol>\n

        The Decision:<\/strong> The context window is for the current step, not the history. Files are your working memory. This allows a 4B parameter model to execute 50-step tasks reliably without losing the plot.<\/p>\n

        \n

        Phase 5: The Architecture \u2014 Brains vs. Brawn<\/h3>\n

        As you scale, mixing compute and data on the same box becomes a liability. If a voice transcription engine spikes while a lawyer is doing RAG, the GPU crashes.<\/p>\n

        The Best Practice: The Two-Box Split<\/h4>\n

        Separate your infrastructure into Stateless Compute<\/strong> and Stateful Orchestration<\/strong>.<\/p>\n

          \n
        • Box A (The Brain – Inference):<\/strong> Pure GPU power. Runs vLLM. Has no database, no user data, no internet access. Holds zero persistent state. If it dies, Ansible rebuilds it in 10 minutes.<\/li>\n
        • Box B (The Body – Application):<\/strong> CPU\/RAM heavy. Runs the UI, Vector DB, RAG pipelines, Guardrails, and Voice processing. Connects to Box A via a private 10GbE crossover cable.<\/li>\n<\/ul>\n

          The Decision:<\/strong> Box A cannot phone home and cannot leak case files. Box B handles all observability and data. You back up Box B obsessively; you don’t back up Box A at all.<\/p>\n

          \n

          Phase 6: Scaling Horizontally<\/h3>\n

          When your firm grows from 5 users to 50, how do you scale?<\/p>\n

          Scaling Throughput (More Users)<\/h4>\n

          Do not buy a bigger GPU. Buy more Box A’s.
          \nRun LiteLLM on Box B as a load balancer. It tracks the active KV cache of vLLM Node 1, Node 2, and Node 3. It routes the new prompt to the least busy GPU.<\/p>\n

          Scaling Capacity (Harder Tasks)<\/h4>\n

          If you need to process 100-page contracts (requiring a massive 128K context window) or run a 70B model, you need Tensor Parallelism.
          \nPut two W7900 GPUs in one chassis. Run vLLM with --tensor-parallel-size 2<\/code>. The model weights split across both GPUs, giving you 96GB of unified VRAM for massive contexts.<\/p>\n

          The Decision:<\/strong> Scale width for concurrency; scale depth (TP) for capacity.<\/p>\n

          \n

          Phase 7: The Voice Stack \u2014 Avoiding the AMD Trap<\/h3>\n

          If you are building on AMD (W7800\/W7900) for cost-efficiency, voice transcription is a landmine. The popular WhisperX<\/code> relies on CTranslate2<\/code>, which is heavily optimized for NVIDIA. On AMD, it is buggy, requires compiling from source, and often forces you to fall back to CPU (killing real-time performance).<\/p>\n

          The Best Practice: whisper.cpp<\/code> + PyAnnote<\/h4>\n

          Bypass Python-centric tooling for C++ native performance.
          \n1. Use whisper.cpp<\/code><\/strong> compiled with HIPBLAS<\/strong> (AMD’s acceleration layer). It is ultra-stable and incredibly fast on ROCm.
          \n2. Run PyAnnote<\/strong> separately (on CPU) to map who<\/em> is speaking (Diarization).
          \n3. Stitch the ultra-fast text output of whisper.cpp<\/code> with the speaker timestamps of PyAnnote in your application layer.<\/p>\n

          The Decision:<\/strong> Use the right tool for the hardware. whisper.cpp<\/code> solves the AMD stability nightmare.<\/p>\n

          \n

          Phase 8: To Fine-Tune or Not to Fine-Tune?<\/h3>\n

          With a new model dropping every month (Qwen 3.5, 3.6, 3.7), spending weeks and thousands of dollars fine-tuning a model feels like chasing the tide.<\/p>\n

          The Rule: RAG for Knowledge, LoRA for Behavior<\/h4>\n