{"id":3405,"date":"2025-05-22T09:07:00","date_gmt":"2025-05-22T15:07:00","guid":{"rendered":"https:\/\/yyc.ninja\/index.php\/2025\/05\/22\/building-an-enterprise-grade-self-hosted-llm-api-for-50-employees-decisions-debugging-and-whats-next\/"},"modified":"2026-10-18T09:29:16","modified_gmt":"2026-10-18T15:29:16","slug":"building-an-enterprise-grade-self-hosted-llm-api-for-50-employees-decisions-debugging-and-whats-next","status":"publish","type":"post","link":"https:\/\/yyc.ninja\/index.php\/2025\/05\/22\/building-an-enterprise-grade-self-hosted-llm-api-for-50-employees-decisions-debugging-and-whats-next\/","title":{"rendered":"Building an Enterprise-Grade Self-Hosted LLM API for 50 Employees: Decisions, Debugging, and What’s Next"},"content":{"rendered":"
\"Building<\/figure>\n

Goal:<\/strong> Serve 50 employees 24\/7 with a production-ready inference API for chat, RAG document Q&A, and coding assistance \u2014 all on-prem or via cost-optimized cloud GPUs, with SSO, RBAC, audit trails, and rock-solid reliability.
\nCurrent state:<\/strong> The Phase 0 reasoner<\/strong> is live: Qwen3-14B-FP8<\/code> running on a Lightning L4 GPU behind vLLM’s OpenAI-compatible server, served through https:\/\/8000-01kq2ztegh41br02eeafs3650f.cloudspaces.litng.ai<\/code>. After a week of YAML, Python, and 10 spicy debugging sessions, we have a working baseline. This is the story of why I built it this way<\/em> \u2014 and where we still need to get better.<\/p>\n

\n

1. The Blueprint: Why vLLM, LiteLLM, and a Moonshot of Sidecars<\/h3>\n

Before a single line of code hit the GPU, the architecture had to answer a hard question: how do you get enterprise-grade availability, security, and auditing out of a single A5000 and a handful of free-software tools?<\/strong><\/p>\n

I chose Branch B<\/strong> \u2014 a single-reasoner deployment (no specialist coder model yet, no multi-node) that leans on:<\/p>\n

    \n
  • vLLM<\/strong> as the inference engine (continuous batching, PagedAttention, auto-prefix caching) <\/li>\n
  • LiteLLM<\/strong> as the gateway (OpenAI-compatible routing, cloud failover, guardrails) <\/li>\n
  • Bifrost<\/strong> (Phase 1) for SSO, RBAC, per-user budgets \u2014 keeping our LiteLLM OSS licence intact <\/li>\n
  • LLM Guard<\/strong> to scrub PII and detect prompt injection on every request <\/li>\n
  • Langfuse<\/strong> + ClickHouse<\/strong> for immutable audit trails and automated quality evals <\/li>\n
  • RAGFlow<\/strong> + Qdrant<\/strong> for document ingestion, vector search, and cross-encoder reranking <\/li>\n<\/ul>\n

    The entire stack fits on the existing Proxmox hypervisor (A5000 24GB) with cloud burst via Lightning.ai \u2014 and, at $803\/month for the L4 while we transition, costs less than a single SaaS seat.<\/p>\n

    \n

    2. Picking the Brain: Qwen3-14B-FP8 and the Quantization Sweet Spot<\/h3>\n

    You can’t serve 50 users from a 3B model without tripping over hallucinations. You also can’t cram a 70B-FP16 into 24 GB of VRAM. The balance point I landed on is Qwen3-14B at FP8<\/strong>.<\/p>\n

      \n
    • Model size matters:<\/strong> Empirically, a 14B cuts hallucination rates nearly in half compared to a 7B, and its reasoning chains stay coherent across multi-step RAG flows. <\/li>\n
    • FP8 is almost lossless:<\/strong> Moving from FP16 to FP8 typically loses only 1-3% quality \u2014 imperceptible to users, yet it halves the VRAM footprint from 28 GB to 14 GB. <\/li>\n
    • Thinking-mode bonus:<\/strong> Qwen3-14B ships with a \/think<\/code> prompt toggle that exposes a chain-of-thought trace. That’s our stand-in for the (unreleased) Qwen3.5-14B-Reasoning model, and it already delivers significant accuracy gains on complex document Q&A.<\/li>\n<\/ul>\n

      When Qwen3.5-14B-Reasoning drops, it’ll be a drop-in HuggingFace swap \u2014 same VRAM budget, better reasoning. And if we ever move to a 48 GB A6000, we can step up to Qwen3-30B-FP8 without changing a single line of orchestration code.<\/p>\n

      \n

      3. The Inference Engine Cage Match: vLLM Wins (For Now)<\/h3>\n

      I evaluated vLLM<\/strong>, SGLang<\/strong>, TensorRT-LLM<\/strong>, and even Ollama<\/strong>. The verdict for a 50-person, always-on business chatbot:<\/p>\n\n\n\n\n\n\n\n
      Engine<\/th>\nWhy not it (today)<\/th>\n<\/tr>\n<\/thead>\n
      SGLang<\/td>\nRadixAttention is jaw-dropping for shared-prefix RAG (3-5\u00d7 faster TTFT), but the ecosystem is still catching up on model support, enterprise hardening, and our team’s tolerance for \u201cbleeding-edge on a Tuesday.\u201d<\/td>\n<\/tr>\n
      TensorRT-LLM<\/td>\nBest raw throughput, but model compilation per update adds a tax we can’t afford at this scale. Rejected unless we hit 200+ users.<\/td>\n<\/tr>\n
      Ollama<\/td>\nWonderful for dev boxes, but no continuous batching means a single chat blocks every other user. Not a production path.<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

      vLLM<\/strong> gives us PagedAttention + chunked prefill + auto prefix caching out of the box, and its OpenAI-compatible \/v1\/chat\/completions<\/code> endpoint means our existing LiteLLM gateway just works.<\/p>\n

      I do have a phase-2 trigger for SGLang<\/strong>: if more than 50% of our RAG requests reuse >4K-token document prefixes, RadixAttention becomes a 3-5\u00d7 latency win that justifies the engine swap. Until then, simplicity wins.<\/p>\n

      \n

      4. The Gateway Tango: LiteLLM OSS + Bifrost<\/h3>\n

      LiteLLM at 10.1.14.82:4000<\/code> is the single-army-knife proxy: it routes chat, embeddings, reranking, and (soon) audio\/file uploads to the right backend, while applying guardrails like PII masking and injection detection.<\/p>\n

      But LiteLLM OSS doesn’t do SSO or per-user budgets. Enter Bifrost<\/strong>, a tiny Go service that sits in front and injects RBAC and spending caps. If we ever upgrade to LiteLLM Enterprise, we can collapse the two layers \u2014 but until then, this pair is clean, auditable, and zero-license-cost.<\/p>\n

      Our failover policy is per-route: general-chat falls back to Gemini Flash Lite on a 5xx, while sensitive document-QA endpoints fail closed with a 503. No noisy data leaves the network unless we explicitly allow it.<\/p>\n

      \n

      5. Security, Observability, and Audit Trails (SOC2-Ready by Design)<\/h3>\n

      From day one, every LLM interaction is:<\/p>\n

        \n
      1. Scanned by LLM Guard<\/strong> \u2014 PII is redacted before<\/em> Langfuse ever sees the prompt. <\/li>\n
      2. Logged immutably in Langfuse<\/strong> \u2014 full trace: user prompt, retrieved vectors, model response, and eval scores. ClickHouse tables are append-only with periodic SHA-256 snapshots to MinIO. <\/li>\n
      3. Monitored by Prometheus + Grafana<\/strong> \u2014 KV cache pressure, TTFT, throughput, and OOM signals.<\/li>\n<\/ol>\n

        We also have a plan for automated LLM-as-a-judge evals<\/strong> inside Langfuse: factual consistency against retrieved documents, PII leakage in output, and rolling quality-degradation alerts. No more manual spot-checking.<\/p>\n

        \n

        6. The Cost Equation: When Cloud Becomes More Expensive Than Hardware<\/h3>\n

        Phase 0<\/strong> runs on a Lightning L4 at $1.10\/hr<\/strong> \u2014 $803\/month. The break-even on our own A5000 24GB<\/strong> (already owned, ~$30\/month electricity) is under 4 months.<\/p>\n

        The long-term hardware map looks like this:<\/p>\n

          \n
        • Phase 1<\/strong> (now): A5000 hosts Qwen3-14B-FP8 + BGE-M3 embeddings + Distil-Whisper ASR sidecars \u2014 total VRAM ~19 GB. <\/li>\n
        • Phase 2<\/strong> upgrade (if quality demands it): 4\u00d7 RTX 5090 workstation (~$15K) to run Qwen3-70B-FP8 with tensor parallelism ~128 GB total VRAM, 800+ TPS. <\/li>\n
        • Phase 3 enterprise<\/strong>: 2\u00d7 used A100 80GB with NVLink for near-linear multi-GPU scaling.<\/li>\n<\/ul>\n

          The trigger to cut over from cloud to on-prem: Lightning spend hits $600 in any single month<\/strong>. Until then, let the cloud carry the burn.<\/p>\n

          \n

          7. RAG Pipeline: RAGFlow, Qdrant, and BGE-M3<\/h3>\n

          Document Q&A is half our use case, so the retrieval pipeline had to be bulletproof:<\/p>\n

            \n
          • RAGFlow<\/strong> ingests PDFs, Word files, Confluence pages, and extracts tables, images, and text. <\/li>\n
          • BGE-M3<\/strong> (568M params, 2.2 GB VRAM) embeds everything into 1024-dim vectors with dense+sparse hybrid search. <\/li>\n
          • Qdrant<\/strong> stores the vectors with RBAC metadata filters so an employee only sees documents they’re allowed to. <\/li>\n
          • BGE-reranker-v2-m3<\/strong> cross-encodes the top-k hits, lifting MRR by 5-10%.<\/li>\n<\/ul>\n

            All of this runs on the same A5000 sidecar, leaving 16+ GB for the reasoner.<\/p>\n

            \n

            8. The Deployment Nightmare: 10 Pitfalls and a Pivot<\/h3>\n

            Bringing up inf-reasoner<\/code> on Lightning’s L4 became a 10-round boxing match. Every time I thought I’d won, a new hook landed \u2014 and each fix is now baked into the commissioning script.<\/p>\n\n\n\n\n\n\n\n\n\n\n\n\n
            Fight #<\/th>\nWhat Went Wrong<\/th>\nRoot Cause<\/th>\nFix<\/th>\n<\/tr>\n<\/thead>\n
            1<\/td>\nStudio went to sleep mid-deploy<\/td>\nLightning’s default auto_sleep<\/code> was 0<\/td>\nSet studio.auto_sleep = False<\/code><\/td>\n<\/tr>\n
            2<\/td>\nresolution-too-deep<\/code> pip explosion<\/td>\nNumpy<2 pin collided with vLLM-0.7<\/td>\nRemoved numpy pin; upgraded scipy first<\/td>\n<\/tr>\n
            3<\/td>\nShell interpreted >=<\/code> as redirect<\/td>\nZsh expanded scipy>=1.13<\/code> into a file write<\/td>\nQuoted all pip specs<\/td>\n<\/tr>\n
            4-4b<\/td>\nnumpy.dtype size changed<\/code> twice<\/td>\nPreinstalled pandas & sklearn built against old numpy ABI<\/td>\nUpgraded scipy<\/code>, scikit-learn<\/code>, pandas<\/code> before installing vLLM<\/td>\n<\/tr>\n
            5<\/td>\nAPI builder plugin failed with unexpected keyword argument<\/code><\/td>\nSDK’s plugin.run()<\/code> didn’t take port<\/code><\/td>\nReplaced with studio.add_ports(8000)<\/code><\/td>\n<\/tr>\n
            6<\/td>\nLitServe rejected enable_async<\/code><\/td>\nIt’s a constructor kwarg, not a class attribute<\/td>\nMoved spec<\/code>, stream<\/code>, enable_async<\/code> into LitAPI(...)<\/code><\/td>\n<\/tr>\n
            7<\/td>\ncannot pickle '_thread.lock'<\/code><\/td>\nLitServe’s spawn tried to pickle the async engine<\/td>\nSet multiprocessing.set_start_method('fork')<\/code><\/td>\n<\/tr>\n
            9<\/td>\nSame pickle error even after fork<\/td>\nLitServe hard-coded spawn<\/code> context at server.py:816<\/code><\/td>\nPivot:<\/strong> abandoned LitServe altogether<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

            The final solution: throw out LitServe and use vLLM’s native OpenAI server directly<\/strong>.
            \npython -m vllm.entrypoints.openai.api_server --model Qwen\/Qwen3-14B-FP8 ...<\/code> gave us \/v1\/chat\/completions<\/code>, embeddings, models, health, and Prometheus metrics \u2014 no proxy, no pickle fights, no ceremony.
            \nI kept the commission.py<\/code> script, but now it writes a run_server.sh<\/code> and detaches vLLM with the exact knobs we need (max_model_len=16384<\/code>, gpu_memory_utilization=0.92<\/code>).<\/p>\n

            The first smoke test returned a streaming completion in 183-ms<\/strong> TTFT. That’s faster than a lot of paid API tiers.<\/p>\n

            \n

            9. What’s Still Missing (And What We’re Doing About It)<\/h3>\n

            Even with a live endpoint, we’re not done. The \u201cdiscussion doc\u201d threw down a gauntlet of enhancements, and I’ve mapped them into a realistic roadmap:<\/p>\n

            Structured Output Enforcement<\/strong> (xgrammar \/ compressed FSM)<\/h4>\n

            For function-calling and JSON responses, hallucinated schemas are a business risk. Will add to L0.7 hardening.<\/p>\n

            Automated Hallucination Detection<\/strong><\/h4>\n

            Langfuse LLM-as-a-judge evals (factual consistency, PII leakage, quality drift) will be configured in L0.6, so we stop relying on \u201csomeone noticed the model was getting weird\u201d.<\/p>\n

            Loop Detection<\/strong><\/h4>\n

            A small n-gram repetition monitor in the callback layer will abort any generation that loops on itself \u2014 something LiteLLM’s 300-s timeout doesn’t catch.<\/p>\n

            Predictive OOM Prevention<\/strong><\/h4>\n

            Instead of screaming after the GPU is full, we’ll track KV cache growth rate and reject requests with 429 once they’d push us past 90% in the next 2 minutes. Combined with TurboQuant’s 3-bit KV compression (4.9\u00d7 memory reduction), OOMs become a solved problem.<\/p>\n

            Multi-modal & Voice<\/strong><\/h4>\n

            The same LiteLLM proxy will route \/v1\/audio\/transcriptions<\/code> to Distil-Whisper large-v3 and \/v1\/audio\/speech<\/code> to Kokoro TTS. File uploads will land in the RAGFlow\/MiniO pipeline. All of this adds <3-GB VRAM and costs pennies per hour.<\/p>\n

            Speculative Decoding<\/strong><\/h4>\n

            Qwen3-1.7B drafts for the reasoner \u2014 a 1.5-2\u00d7 decode speedup on interactive traffic, enabled with a single vLLM flag. Disabled for batch workloads, because there it hurts throughput.<\/p>\n

            Disaggregated Prefill\/Decode<\/strong> (Phase 3)<\/h4>\n

            Once we have a second A5000, we can dedicate one GPU to prompt processing and the other to token generation, further slashing TTFT on long-context RAG.<\/p>\n

            \n

            10. Reflections: The Biggest Lessons from This Build<\/h3>\n
              \n
            1. Start with the simplest engine, not the fastest.<\/strong> vLLM’s native server saved days of fighting framework mismatches. SGLang is waiting, but only when the data says we need it. <\/li>\n
            2. Quantization is a multiplier, not a crime.<\/strong> FP8 gave us a 14B model in 14-GB with imperceptible quality loss. The \u201cnever quantize\u201d crowd hasn’t run a 24\/7 service on $800\/month. <\/li>\n
            3. The gateway is the backbone.<\/strong> LiteLLM plus a lightweight auth proxy (Bifrost) made RBAC, failover, and guardrails plug-and-play. I can’t imagine managing 50 users without a central traffic cop. <\/li>\n
            4. Deployment is where the rubber burns.<\/strong> Ten consecutive failures in one commissioning run could’ve broken a less stubborn spirit. Now every fix is automated, and the next clean run will take 15 minutes from start to live. <\/li>\n
            5. Cost is a forcing function.<\/strong> The $600\/month threshold to cut over from cloud to on-prem keeps us honest \u2014 and proves that self-hosted LLMs are not just for hyperscalers.<\/li>\n<\/ol>\n
              \n

              What’s Live Right Now<\/h3>\n
              curl https:\/\/8000-01kq2ztegh41br02eeafs3650f.cloudspaces.litng.ai\/v1\/chat\/completions \\\n  -H "Authorization: Bearer litserve-reasoner-key" \\\n  -H "Content-Type: application\/json" \\\n  -d '{"model":"Qwen\/Qwen3-14B-FP8","messages":[{"role":"user","content":"What is the capital of France?"}]}'\n<\/code><\/pre>\n
              (First token in ~183-ms, full response in a blink.)<\/p>\n
              The path from \u201cI have a GPU and a dream\u201d to \u201cI have a live, monitored, production-grade API\u201d is paved with tiny, correct decisions and a willingness to throw away the parts that fight back. The next chapter: structured outputs, automated quality gates, and a world where 14B-FP8 is just the minimum<\/em> bar.<\/p>\n\n\n
              \n\n\n\n
              Get the next one in your inbox<\/h2>\n\n\n\n
              Enjoyed this? I write about local AI, self-hosting, and cloud cost ninjutsu. Join the Ninja Dispatch — occasional, high-signal, no fluff.<\/p>\n\n\n\n\t\t\t