Goal: Serve 50 employees 24/7 with a production-ready inference API for chat, RAG document Q&A, and coding assistance — all on-prem or via cost-optimized cloud GPUs, with SSO, RBAC, audit trails, and rock-solid reliability.
Current state: The Phase 0 reasoner is live: Qwen3-14B-FP8 running on a Lightning L4 GPU behind vLLM’s OpenAI-compatible server, served through https://8000-01kq2ztegh41br02eeafs3650f.cloudspaces.litng.ai. After a week of YAML, Python, and 10 spicy debugging sessions, we have a working baseline. This is the story of why I built it this way — and where we still need to get better.
1. The Blueprint: Why vLLM, LiteLLM, and a Moonshot of Sidecars
Before a single line of code hit the GPU, the architecture had to answer a hard question: how do you get enterprise-grade availability, security, and auditing out of a single A5000 and a handful of free-software tools?
I chose Branch B — a single-reasoner deployment (no specialist coder model yet, no multi-node) that leans on:
- vLLM as the inference engine (continuous batching, PagedAttention, auto-prefix caching)
- LiteLLM as the gateway (OpenAI-compatible routing, cloud failover, guardrails)
- Bifrost (Phase 1) for SSO, RBAC, per-user budgets — keeping our LiteLLM OSS licence intact
- LLM Guard to scrub PII and detect prompt injection on every request
- Langfuse + ClickHouse for immutable audit trails and automated quality evals
- RAGFlow + Qdrant for document ingestion, vector search, and cross-encoder reranking
The entire stack fits on the existing Proxmox hypervisor (A5000 24GB) with cloud burst via Lightning.ai — and, at $803/month for the L4 while we transition, costs less than a single SaaS seat.
2. Picking the Brain: Qwen3-14B-FP8 and the Quantization Sweet Spot
You can’t serve 50 users from a 3B model without tripping over hallucinations. You also can’t cram a 70B-FP16 into 24 GB of VRAM. The balance point I landed on is Qwen3-14B at FP8.
- Model size matters: Empirically, a 14B cuts hallucination rates nearly in half compared to a 7B, and its reasoning chains stay coherent across multi-step RAG flows.
- FP8 is almost lossless: Moving from FP16 to FP8 typically loses only 1-3% quality — imperceptible to users, yet it halves the VRAM footprint from 28 GB to 14 GB.
- Thinking-mode bonus: Qwen3-14B ships with a
/thinkprompt toggle that exposes a chain-of-thought trace. That’s our stand-in for the (unreleased) Qwen3.5-14B-Reasoning model, and it already delivers significant accuracy gains on complex document Q&A.
When Qwen3.5-14B-Reasoning drops, it’ll be a drop-in HuggingFace swap — same VRAM budget, better reasoning. And if we ever move to a 48 GB A6000, we can step up to Qwen3-30B-FP8 without changing a single line of orchestration code.
3. The Inference Engine Cage Match: vLLM Wins (For Now)
I evaluated vLLM, SGLang, TensorRT-LLM, and even Ollama. The verdict for a 50-person, always-on business chatbot:
| Engine | Why not it (today) |
|---|---|
| SGLang | RadixAttention is jaw-dropping for shared-prefix RAG (3-5× faster TTFT), but the ecosystem is still catching up on model support, enterprise hardening, and our team’s tolerance for “bleeding-edge on a Tuesday.” |
| TensorRT-LLM | Best raw throughput, but model compilation per update adds a tax we can’t afford at this scale. Rejected unless we hit 200+ users. |
| Ollama | Wonderful for dev boxes, but no continuous batching means a single chat blocks every other user. Not a production path. |
vLLM gives us PagedAttention + chunked prefill + auto prefix caching out of the box, and its OpenAI-compatible /v1/chat/completions endpoint means our existing LiteLLM gateway just works.
I do have a phase-2 trigger for SGLang: if more than 50% of our RAG requests reuse >4K-token document prefixes, RadixAttention becomes a 3-5× latency win that justifies the engine swap. Until then, simplicity wins.
4. The Gateway Tango: LiteLLM OSS + Bifrost
LiteLLM at 10.1.14.82:4000 is the single-army-knife proxy: it routes chat, embeddings, reranking, and (soon) audio/file uploads to the right backend, while applying guardrails like PII masking and injection detection.
But LiteLLM OSS doesn’t do SSO or per-user budgets. Enter Bifrost, a tiny Go service that sits in front and injects RBAC and spending caps. If we ever upgrade to LiteLLM Enterprise, we can collapse the two layers — but until then, this pair is clean, auditable, and zero-license-cost.
Our failover policy is per-route: general-chat falls back to Gemini Flash Lite on a 5xx, while sensitive document-QA endpoints fail closed with a 503. No noisy data leaves the network unless we explicitly allow it.
5. Security, Observability, and Audit Trails (SOC2-Ready by Design)
From day one, every LLM interaction is:
- Scanned by LLM Guard — PII is redacted before Langfuse ever sees the prompt.
- Logged immutably in Langfuse — full trace: user prompt, retrieved vectors, model response, and eval scores. ClickHouse tables are append-only with periodic SHA-256 snapshots to MinIO.
- Monitored by Prometheus + Grafana — KV cache pressure, TTFT, throughput, and OOM signals.
We also have a plan for automated LLM-as-a-judge evals inside Langfuse: factual consistency against retrieved documents, PII leakage in output, and rolling quality-degradation alerts. No more manual spot-checking.
6. The Cost Equation: When Cloud Becomes More Expensive Than Hardware
Phase 0 runs on a Lightning L4 at $1.10/hr — $803/month. The break-even on our own A5000 24GB (already owned, ~$30/month electricity) is under 4 months.
The long-term hardware map looks like this:
- Phase 1 (now): A5000 hosts Qwen3-14B-FP8 + BGE-M3 embeddings + Distil-Whisper ASR sidecars — total VRAM ~19 GB.
- Phase 2 upgrade (if quality demands it): 4× RTX 5090 workstation (~$15K) to run Qwen3-70B-FP8 with tensor parallelism ~128 GB total VRAM, 800+ TPS.
- Phase 3 enterprise: 2× used A100 80GB with NVLink for near-linear multi-GPU scaling.
The trigger to cut over from cloud to on-prem: Lightning spend hits $600 in any single month. Until then, let the cloud carry the burn.
7. RAG Pipeline: RAGFlow, Qdrant, and BGE-M3
Document Q&A is half our use case, so the retrieval pipeline had to be bulletproof:
- RAGFlow ingests PDFs, Word files, Confluence pages, and extracts tables, images, and text.
- BGE-M3 (568M params, 2.2 GB VRAM) embeds everything into 1024-dim vectors with dense+sparse hybrid search.
- Qdrant stores the vectors with RBAC metadata filters so an employee only sees documents they’re allowed to.
- BGE-reranker-v2-m3 cross-encodes the top-k hits, lifting MRR by 5-10%.
All of this runs on the same A5000 sidecar, leaving 16+ GB for the reasoner.
8. The Deployment Nightmare: 10 Pitfalls and a Pivot
Bringing up inf-reasoner on Lightning’s L4 became a 10-round boxing match. Every time I thought I’d won, a new hook landed — and each fix is now baked into the commissioning script.
| Fight # | What Went Wrong | Root Cause | Fix |
|---|---|---|---|
| 1 | Studio went to sleep mid-deploy | Lightning’s default auto_sleep was 0 |
Set studio.auto_sleep = False |
| 2 | resolution-too-deep pip explosion |
Numpy<2 pin collided with vLLM-0.7 | Removed numpy pin; upgraded scipy first |
| 3 | Shell interpreted >= as redirect |
Zsh expanded scipy>=1.13 into a file write |
Quoted all pip specs |
| 4-4b | numpy.dtype size changed twice |
Preinstalled pandas & sklearn built against old numpy ABI | Upgraded scipy, scikit-learn, pandas before installing vLLM |
| 5 | API builder plugin failed with unexpected keyword argument |
SDK’s plugin.run() didn’t take port |
Replaced with studio.add_ports(8000) |
| 6 | LitServe rejected enable_async |
It’s a constructor kwarg, not a class attribute | Moved spec, stream, enable_async into LitAPI(...) |
| 7 | cannot pickle '_thread.lock' |
LitServe’s spawn tried to pickle the async engine | Set multiprocessing.set_start_method('fork') |
| 9 | Same pickle error even after fork | LitServe hard-coded spawn context at server.py:816 |
Pivot: abandoned LitServe altogether |
The final solution: throw out LitServe and use vLLM’s native OpenAI server directly.
python -m vllm.entrypoints.openai.api_server --model Qwen/Qwen3-14B-FP8 ... gave us /v1/chat/completions, embeddings, models, health, and Prometheus metrics — no proxy, no pickle fights, no ceremony.
I kept the commission.py script, but now it writes a run_server.sh and detaches vLLM with the exact knobs we need (max_model_len=16384, gpu_memory_utilization=0.92).
The first smoke test returned a streaming completion in 183-ms TTFT. That’s faster than a lot of paid API tiers.
9. What’s Still Missing (And What We’re Doing About It)
Even with a live endpoint, we’re not done. The “discussion doc” threw down a gauntlet of enhancements, and I’ve mapped them into a realistic roadmap:
Structured Output Enforcement (xgrammar / compressed FSM)
For function-calling and JSON responses, hallucinated schemas are a business risk. Will add to L0.7 hardening.
Automated Hallucination Detection
Langfuse LLM-as-a-judge evals (factual consistency, PII leakage, quality drift) will be configured in L0.6, so we stop relying on “someone noticed the model was getting weird”.
Loop Detection
A small n-gram repetition monitor in the callback layer will abort any generation that loops on itself — something LiteLLM’s 300-s timeout doesn’t catch.
Predictive OOM Prevention
Instead of screaming after the GPU is full, we’ll track KV cache growth rate and reject requests with 429 once they’d push us past 90% in the next 2 minutes. Combined with TurboQuant’s 3-bit KV compression (4.9× memory reduction), OOMs become a solved problem.
Multi-modal & Voice
The same LiteLLM proxy will route /v1/audio/transcriptions to Distil-Whisper large-v3 and /v1/audio/speech to Kokoro TTS. File uploads will land in the RAGFlow/MiniO pipeline. All of this adds <3-GB VRAM and costs pennies per hour.
Speculative Decoding
Qwen3-1.7B drafts for the reasoner — a 1.5-2× decode speedup on interactive traffic, enabled with a single vLLM flag. Disabled for batch workloads, because there it hurts throughput.
Disaggregated Prefill/Decode (Phase 3)
Once we have a second A5000, we can dedicate one GPU to prompt processing and the other to token generation, further slashing TTFT on long-context RAG.
10. Reflections: The Biggest Lessons from This Build
- Start with the simplest engine, not the fastest. vLLM’s native server saved days of fighting framework mismatches. SGLang is waiting, but only when the data says we need it.
- Quantization is a multiplier, not a crime. FP8 gave us a 14B model in 14-GB with imperceptible quality loss. The “never quantize” crowd hasn’t run a 24/7 service on $800/month.
- The gateway is the backbone. LiteLLM plus a lightweight auth proxy (Bifrost) made RBAC, failover, and guardrails plug-and-play. I can’t imagine managing 50 users without a central traffic cop.
- Deployment is where the rubber burns. Ten consecutive failures in one commissioning run could’ve broken a less stubborn spirit. Now every fix is automated, and the next clean run will take 15 minutes from start to live.
- Cost is a forcing function. The $600/month threshold to cut over from cloud to on-prem keeps us honest — and proves that self-hosted LLMs are not just for hyperscalers.
What’s Live Right Now
curl https://8000-01kq2ztegh41br02eeafs3650f.cloudspaces.litng.ai/v1/chat/completions \
-H "Authorization: Bearer litserve-reasoner-key" \
-H "Content-Type: application/json" \
-d '{"model":"Qwen/Qwen3-14B-FP8","messages":[{"role":"user","content":"What is the capital of France?"}]}'
(First token in ~183-ms, full response in a blink.)
The path from “I have a GPU and a dream” to “I have a live, monitored, production-grade API” is paved with tiny, correct decisions and a willingness to throw away the parts that fight back. The next chapter: structured outputs, automated quality gates, and a world where 14B-FP8 is just the minimum bar.
Get the next one in your inbox
Enjoyed this? I write about local AI, self-hosting, and cloud cost ninjutsu. Join the Ninja Dispatch — occasional, high-signal, no fluff.
