{"id":3423,"date":"2022-09-13T15:13:00","date_gmt":"2022-09-13T21:13:00","guid":{"rendered":"https:\/\/yyc.ninja\/index.php\/2022\/09\/13\/how-to-build-big-search-a-self-hosted-hybrid-ai-search-platform-over-your-file-shares\/"},"modified":"2026-10-19T12:18:08","modified_gmt":"2026-10-19T18:18:08","slug":"how-to-build-big-search-a-self-hosted-hybrid-ai-search-platform-over-your-file-shares","status":"publish","type":"post","link":"https:\/\/yyc.ninja\/index.php\/2022\/09\/13\/how-to-build-big-search-a-self-hosted-hybrid-ai-search-platform-over-your-file-shares\/","title":{"rendered":"How to build &#8220;big-search&#8221;: a self-hosted hybrid AI search platform over your file shares"},"content":{"rendered":"<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1200\" height=\"630\" src=\"https:\/\/yyc.ninja\/wp-content\/uploads\/2026\/10\/how-to-build-big-search-a-self-hosted-hybrid-ai-search-platf-cover-logo.png\" alt=\"How to build \"big-search\": a self-hosted hybrid AI search platform over your file shares\" class=\"wp-image-3475\" sizes=\"auto, (max-width: 1200px) 100vw, 1200px\" srcset=\"https:\/\/yyc.ninja\/wp-content\/uploads\/2026\/10\/how-to-build-big-search-a-self-hosted-hybrid-ai-search-platf-cover-logo.png 1200w, https:\/\/yyc.ninja\/wp-content\/uploads\/2026\/10\/how-to-build-big-search-a-self-hosted-hybrid-ai-search-platf-cover-logo-300x158.png 300w, https:\/\/yyc.ninja\/wp-content\/uploads\/2026\/10\/how-to-build-big-search-a-self-hosted-hybrid-ai-search-platf-cover-logo-1024x538.png 1024w, https:\/\/yyc.ninja\/wp-content\/uploads\/2026\/10\/how-to-build-big-search-a-self-hosted-hybrid-ai-search-platf-cover-logo-768x403.png 768w, https:\/\/yyc.ninja\/wp-content\/uploads\/2026\/10\/how-to-build-big-search-a-self-hosted-hybrid-ai-search-platf-cover-logo-100x53.png 100w\" \/><\/figure>\n<p class=\"wp-block-paragraph\"><strong>Audience:<\/strong> Someone with a NAS full of documents, a couple of Linux boxes, and an itch to make &#8220;search my stuff&#8221; actually work, including semantic search, OCR, and an LLM-powered Q&amp;A layer.<\/p>\n<p class=\"wp-block-paragraph\">This is the build log for a working system as of 2026-05-28. ~32,000 documents indexed across personal documents, course materials, code, and a PDF library. Hybrid BM25 + k-NN semantic retrieval with Reciprocal Rank Fusion, OCR for images and scanned PDFs, and a chat-style Q&amp;A agent on top.<\/p>\n<hr class=\"wp-block-separator\" \/>\n<h3 class=\"wp-block-heading\">0. What you end up with<\/h3>\n<pre><code>You \u2500\u2500\u25ba Dify chat agent\n         \u2502\n         \u25bc POST \/search\/portable  (or \/search\/airflow)\n       search-api  (FastAPI on :8088)\n         \u2502\n         \u25bc hybrid: BM25 + k-NN, fused with RRF\n       OpenSearch 2.19.5  (:9200)\n         \u2502\n         \u251c\u2500\u25ba Faiss HNSW index over content_vector\n         \u251c\u2500\u25ba all-MiniLM-L6-v2 embedding model (in-cluster, 384-dim)\n         \u2514\u2500\u25ba nlp-pipeline (ingest)\n         \u25b2\n         \u2502 _bulk\n       FSCrawler  \u00d7N  (one per source)\n         \u2502\n         \u25bc reads files\n       NAS via NFSv4\n<\/code><\/pre>\n<p class=\"wp-block-paragraph\">Plus monitoring (Prometheus + Grafana + Loki), nightly snapshot backups to the NAS, and an LLM gateway (LiteLLM) for unified embedding\/completion access.<\/p>\n<hr class=\"wp-block-separator\" \/>\n<h3 class=\"wp-block-heading\">1. The pieces and why each one<\/h3>\n<h4 class=\"wp-block-heading\">1.1 OpenSearch (not Elasticsearch)<\/h4>\n<p class=\"wp-block-paragraph\">Apache-2.0 licensed fork of Elasticsearch 7.10. We picked it for two specific reasons:<\/p>\n<ul class=\"wp-block-list\">\n<li><strong><code>neural_search<\/code> plugin ships out of the box.<\/strong> No vector DB to deploy separately. Embeddings live next to the BM25 index.<\/li>\n<li><strong><code>hybrid<\/code> query with <code>score-ranker-processor<\/code><\/strong> (added in OS 2.17) supports true Reciprocal Rank Fusion. Hybrid keyword + semantic is one query, not two.<\/li>\n<\/ul>\n<p class=\"wp-block-paragraph\">Started on 2.11, upgraded to 2.19.5 mid-project for RRF and inline k-NN filters. Lesson: pick 2.19+ from day one if you can.<\/p>\n<h4 class=\"wp-block-heading\">1.2 FSCrawler<\/h4>\n<p class=\"wp-block-paragraph\">Crawls a filesystem, runs Apache Tika on each file, and <code>_bulk<\/code>-writes the extracted text into OpenSearch. Handles:<\/p>\n<ul class=\"wp-block-list\">\n<li><strong>Text formats<\/strong> (Markdown, code, JSON, YAML, HTML, plain text).<\/li>\n<li><strong>Office formats<\/strong> (Word, Excel, PowerPoint) via Tika.<\/li>\n<li><strong>PDFs<\/strong> with two strategies: <code>text<\/code> (fast, only extracts already-embedded text) and <code>ocr_and_text<\/code> (runs Tesseract on image-only pages). We use <code>ocr_and_text<\/code>.<\/li>\n<li><strong>Images<\/strong> (<code>.png<\/code>, <code>.jpg<\/code>, <code>.webp<\/code>, etc.): Tika invokes Tesseract for OCR.<\/li>\n<li><strong>Subtitles<\/strong> (<code>.srt<\/code>, <code>.vtt<\/code>): treated as plain text once Tika MIME detection is configured.<\/li>\n<\/ul>\n<p class=\"wp-block-paragraph\">One FSCrawler container per data source so each can have its own scan schedule and <code>excludes<\/code> list. We run one for <code>\/portable\/<\/code>, one for <code>\/airflow\/<\/code> on the NAS. Each writes to its own index.<\/p>\n<h4 class=\"wp-block-heading\">1.3 Apache Tika (inside FSCrawler)<\/h4>\n<p class=\"wp-block-paragraph\">Format detector + text extractor. The thing that makes a PDF and a <code>.docx<\/code> and a <code>.png<\/code> all look the same to the indexer downstream. Tika&#8217;s MIME detection is <em>almost<\/em> good enough out of the box. The gaps we hit:<\/p>\n<ul class=\"wp-block-list\">\n<li><code>.srt<\/code> was being parsed but not always classified as text, fixed with an explicit MIME override.<\/li>\n<li>Image-only PDFs need OCR enabled (<code>pdf_strategy: ocr_and_text<\/code>).<\/li>\n<li>Standalone images need <code>ocr.enabled: true<\/code> (default is on, but worth verifying).<\/li>\n<\/ul>\n<h4 class=\"wp-block-heading\">1.4 The embedding model: <code>all-MiniLM-L6-v2<\/code><\/h4>\n<p class=\"wp-block-paragraph\">384-dimensional sentence-transformer, runs on CPU inside OpenSearch via the <code>_plugins\/_ml<\/code> API. Picked because:<\/p>\n<ul class=\"wp-block-list\">\n<li>Cheap to run (~5 ms per query on a laptop CPU).<\/li>\n<li>Good enough quality for retrieval-grade similarity.<\/li>\n<li>Fits in memory next to the search workload.<\/li>\n<\/ul>\n<p class=\"wp-block-paragraph\">Downside: 384-dim is small. If quality plateaus, swap to <code>nomic-embed-text<\/code> (768-dim, also self-hostable on Ollama), we already have it wired into LiteLLM for the next iteration.<\/p>\n<h4 class=\"wp-block-heading\">1.5 Faiss + HNSW (the k-NN engine)<\/h4>\n<p class=\"wp-block-paragraph\">OpenSearch ships three k-NN engines: Lucene, NMSLIB, Faiss. We started on NMSLIB (the OS 2.11 default), hit a wall on filtered k-NN queries (NMSLIB doesn&#8217;t support them), and migrated all three indices to Faiss. With Faiss:<\/p>\n<ul class=\"wp-block-list\">\n<li>The <code>filter<\/code> clause on a <code>neural<\/code> sub-query is pushed into the HNSW graph traversal.<\/li>\n<li>A query for &#8220;Markdown files about Kubernetes&#8221; returns documents that are <em>both<\/em> semantically about Kubernetes <em>and<\/em> actually <code>.md<\/code>, no client-side post-pruning.<\/li>\n<\/ul>\n<p class=\"wp-block-paragraph\">Trade-off: Faiss requires reindexing the whole field. Plan for it from day one.<\/p>\n<h4 class=\"wp-block-heading\">1.6 search-api (FastAPI)<\/h4>\n<p class=\"wp-block-paragraph\">Thin Python service that:<\/p>\n<ul class=\"wp-block-list\">\n<li>Accepts <code>{query, top_k, filters}<\/code> from clients.<\/li>\n<li>Builds the OpenSearch hybrid query JSON (one <code>match<\/code> clause + one <code>neural<\/code> clause, fused by the RRF pipeline).<\/li>\n<li>Returns the top-k hits with the document content and metadata.<\/li>\n<\/ul>\n<p class=\"wp-block-paragraph\">Why a service in front of OpenSearch instead of letting Dify hit OpenSearch directly: it normalizes filter syntax (callers can send <code>filters<\/code> as an object, a JSON string, or a Python <code>repr<\/code>, all three coerce correctly), and it lets us swap the underlying engine without touching every client.<\/p>\n<h4 class=\"wp-block-heading\">1.7 Dify (the chat agent layer)<\/h4>\n<p class=\"wp-block-paragraph\">Open-source LLM application platform. Each chat flow is a graph of nodes: router LLM \u2192 HTTP tool (call <code>search-api<\/code>) \u2192 answerer LLM. We use it instead of writing custom RAG code because the visual editor makes it trivial to change the prompt or swap the LLM. The agent flow that&#8217;s live calls <code>POST \/search<\/code> with filters chosen by the router LLM, then synthesizes an answer with citations to <code>path.virtual<\/code>.<\/p>\n<h4 class=\"wp-block-heading\">1.8 LiteLLM (the LLM gateway)<\/h4>\n<p class=\"wp-block-paragraph\">OpenAI-compatible proxy in front of every model we use: Ollama (local), Anthropic (cloud), OpenAI (cloud), Nomic embedding. Means every code path that wants an LLM speaks one dialect, and we get unified request logging + cost tracking for free.<\/p>\n<h4 class=\"wp-block-heading\">1.9 Monitoring (the unglamorous foundation)<\/h4>\n<ul class=\"wp-block-list\">\n<li><strong>Prometheus<\/strong> scrapes:<\/li>\n<li><code>opensearch-exporter<\/code> (cluster, JVM, index metrics)<\/li>\n<li><code>bigsearch_metrics<\/code>: a custom Python sidecar that emits 11 domain signals (<code>bigsearch_ml_model_deployed<\/code>, <code>bigsearch_embed_latency_seconds<\/code>, <code>bigsearch_index_with_vector_ratio<\/code>, <code>bigsearch_snapshot_age_seconds<\/code>, <code>bigsearch_nas_reachable<\/code>, etc.)<\/li>\n<li><code>nas-mount-monitor<\/code>: emits 0\/1 if NFS mounts are alive<\/li>\n<li><strong>Grafana<\/strong> dashboard <code>bigsearch-overview<\/code> shows it all.<\/li>\n<li><strong>Loki + Alloy<\/strong> ship container logs.<\/li>\n<li><strong>13 alert rules<\/strong> cover: cluster red, model not deployed, NAS unreachable, snapshot too old, embedding-ratio drift, search-api down.<\/li>\n<\/ul>\n<p class=\"wp-block-paragraph\">You learn the value of monitoring the day FSCrawler silently deletes 276,000 documents because an NFS mount unmounted itself. Build the alerts first.<\/p>\n<hr class=\"wp-block-separator\" \/>\n<h3 class=\"wp-block-heading\">2. The build, in order<\/h3>\n<h4 class=\"wp-block-heading\">Step 1: bring up OpenSearch<\/h4>\n<p class=\"wp-block-paragraph\"><code>docker compose up -d opensearch-node1 opensearch-dashboards<\/code>. Single node, security plugin disabled (it&#8217;s on a LAN). 8 GiB heap to start.<\/p>\n<p class=\"wp-block-paragraph\">Verify with <code>GET \/_cluster\/health<\/code> \u2192 <code>green<\/code>. Open Dashboards on <code>:5601<\/code>.<\/p>\n<h4 class=\"wp-block-heading\">Step 2: deploy the ML model<\/h4>\n<pre><code class=\"language-bash\">curl -X POST &quot;http:\/\/localhost:9200\/_plugins\/_ml\/models\/_register&quot; \\\n  -H 'Content-Type: application\/json' \\\n  -d '{\n    &quot;name&quot;: &quot;huggingface\/sentence-transformers\/all-MiniLM-L6-v2&quot;,\n    &quot;version&quot;: &quot;1.0.1&quot;,\n    &quot;model_format&quot;: &quot;TORCH_SCRIPT&quot;\n  }'\n<\/code><\/pre>\n<p class=\"wp-block-paragraph\">Then <code>_deploy<\/code> the returned model_id. The model lives inside OpenSearch from this point on. Verify by predicting a one-sentence embedding (should return 384 floats).<\/p>\n<p class=\"wp-block-paragraph\"><strong>Gotcha we hit:<\/strong> an OS 2.11 \u2192 2.19 upgrade left a stale <code>cpu-precxx11<\/code> PyTorch cache whose <code>libstdc++<\/code> lacked <code>CXXABI_1.3.9<\/code>. Model deployment threw <code>UnsatisfiedLinkError<\/code>. Fix: set <code>PYTORCH_PRECXX11=false<\/code> env var on the OpenSearch container, clear <code>data\/ml_cache\/pytorch<\/code>, recreate, redeploy.<\/p>\n<h4 class=\"wp-block-heading\">Step 3: create the ingest pipeline<\/h4>\n<pre><code class=\"language-bash\">curl -X PUT &quot;http:\/\/localhost:9200\/_ingest\/pipeline\/nlp-pipeline&quot; \\\n  -H 'Content-Type: application\/json' \\\n  -d '{\n    &quot;processors&quot;: [{\n      &quot;text_embedding&quot;: {\n        &quot;model_id&quot;: &quot;BST8XJwBu6n7bcim86zF&quot;,\n        &quot;field_map&quot;: {&quot;content&quot;: &quot;content_vector&quot;}\n      }\n    }]\n  }'\n<\/code><\/pre>\n<p class=\"wp-block-paragraph\">Now every document indexed with <code>?pipeline=nlp-pipeline<\/code> gets a <code>content_vector<\/code> field populated by the model.<\/p>\n<h4 class=\"wp-block-heading\">Step 4: create the index mapping (Faiss from day one)<\/h4>\n<p class=\"wp-block-paragraph\">The critical bit:<\/p>\n<pre><code class=\"language-json\">{\n  &quot;mappings&quot;: {\n    &quot;properties&quot;: {\n      &quot;content&quot;: {&quot;type&quot;: &quot;text&quot;},\n      &quot;content_vector&quot;: {\n        &quot;type&quot;: &quot;knn_vector&quot;,\n        &quot;dimension&quot;: 384,\n        &quot;method&quot;: {\n          &quot;name&quot;: &quot;hnsw&quot;,\n          &quot;engine&quot;: &quot;faiss&quot;,\n          &quot;space_type&quot;: &quot;l2&quot;,\n          &quot;parameters&quot;: {&quot;ef_construction&quot;: 256, &quot;m&quot;: 16}\n        }\n      }\n    }\n  },\n  &quot;settings&quot;: {\n    &quot;index.knn&quot;: true,\n    &quot;default_pipeline&quot;: &quot;nlp-pipeline&quot;\n  }\n}\n<\/code><\/pre>\n<p class=\"wp-block-paragraph\"><code>engine: faiss<\/code> is what enables filtered k-NN later. Don&#8217;t skip it.<\/p>\n<h4 class=\"wp-block-heading\">Step 5: the hybrid search pipeline<\/h4>\n<pre><code class=\"language-bash\">curl -X PUT &quot;http:\/\/localhost:9200\/_search\/pipeline\/hybrid-rrf-pipeline&quot; \\\n  -H 'Content-Type: application\/json' \\\n  -d '{\n    &quot;phase_results_processors&quot;: [{\n      &quot;score-ranker-processor&quot;: {\n        &quot;combination&quot;: {&quot;technique&quot;: &quot;rrf&quot;, &quot;rank_constant&quot;: 60}\n      }\n    }]\n  }'\n<\/code><\/pre>\n<p class=\"wp-block-paragraph\">This is how BM25 and k-NN get fused into one ranked list.<\/p>\n<h4 class=\"wp-block-heading\">Step 6: bring up FSCrawler<\/h4>\n<p class=\"wp-block-paragraph\"><code>_settings.json<\/code> per crawler. Key fields:<\/p>\n<pre><code class=\"language-json\">{\n  &quot;fs&quot;: {\n    &quot;url&quot;: &quot;\/tmp\/es\/portable&quot;,\n    &quot;update_rate&quot;: &quot;15m&quot;,\n    &quot;remove_deleted&quot;: false,\n    &quot;continue_on_error&quot;: true,\n    &quot;ocr&quot;: {&quot;enabled&quot;: true, &quot;pdf_strategy&quot;: &quot;ocr_and_text&quot;, &quot;language&quot;: &quot;eng&quot;},\n    &quot;excludes&quot;: [&quot;*\/node_modules\/*&quot;, &quot;*\/.git\/*&quot;, &quot;*\/__pycache__\/*&quot;, ...]\n  },\n  &quot;elasticsearch&quot;: {\n    &quot;urls&quot;: [&quot;http:\/\/opensearch-node1:9200&quot;],\n    &quot;index&quot;: &quot;portable_docs&quot;,\n    &quot;pipeline&quot;: &quot;nlp-pipeline&quot;\n  }\n}\n<\/code><\/pre>\n<p class=\"wp-block-paragraph\"><strong><code>remove_deleted: false<\/code><\/strong> is the most important line in the entire system. The default (<code>true<\/code>) means FSCrawler will delete documents from OpenSearch if it can&#8217;t find the source file. If your NFS mount drops, you lose your index. We learned this the hard way (incident report linked at the end).<\/p>\n<h4 class=\"wp-block-heading\">Step 7: search-api<\/h4>\n<p class=\"wp-block-paragraph\">FastAPI service. ~150 lines. Builds and proxies the hybrid query, normalizes filter inputs. The whole point is to have one endpoint that the Dify agent (and any other consumer) can hit.<\/p>\n<h4 class=\"wp-block-heading\">Step 8: Dify<\/h4>\n<p class=\"wp-block-paragraph\">Wire a custom HTTP tool against <code>POST \/search<\/code>. Build a flow: <code>Start \u2192 Router LLM (pick doc_type filter) \u2192 HTTP tool \u2192 Answerer LLM (cite path.virtual) \u2192 End<\/code>. Single-shot (no re-search loop) but enough to validate the chat \u2192 retrieval \u2192 LLM loop end-to-end.<\/p>\n<h4 class=\"wp-block-heading\">Step 9: monitoring and snapshots<\/h4>\n<ul class=\"wp-block-list\">\n<li>Set up the snapshot repo pointing at a NAS path. Run a daily cron that calls <code>_snapshot\/...\/snapshot_$(date +%Y%m%d)?wait_for_completion=true<\/code>.<\/li>\n<li>Deploy the Prometheus exporter sidecar.<\/li>\n<li>Write <code>bigsearch_metrics.py<\/code> to emit the things the off-the-shelf exporter doesn&#8217;t: ML model state, embedding coverage ratio, NFS mount status.<\/li>\n<li>Load the Grafana dashboard JSON.<\/li>\n<li>Configure 5-10 alert rules. Start with: cluster red, model not deployed, snapshot age &gt; 36 h, embedding ratio drops &gt; 5%, NFS mount missing.<\/li>\n<\/ul>\n<hr class=\"wp-block-separator\" \/>\n<h3 class=\"wp-block-heading\">3. Lessons that cost real time<\/h3>\n<h4 class=\"wp-block-heading\"><code>remove_deleted: true<\/code> is a footgun<\/h4>\n<p class=\"wp-block-paragraph\">When NFS mounts dropped, FSCrawler walked the (now-empty) tree, decided 276,000 documents had been &#8220;deleted,&#8221; and removed them all from the index. The fix is one line. Set it before you have a corpus to lose.<\/p>\n<h4 class=\"wp-block-heading\">NMSLIB \u2192 Faiss is non-negotiable<\/h4>\n<p class=\"wp-block-paragraph\">Inline k-NN filters are the difference between &#8220;semantic search that returns relevant results&#8221; and &#8220;semantic search that returns whatever its closest neighbours are, ignoring your filters.&#8221; Pick Faiss at index-creation time. Reindexing 30,000 docs later isn&#8217;t catastrophic, but it&#8217;s hours you don&#8217;t need to spend.<\/p>\n<h4 class=\"wp-block-heading\">Hybrid scores are tiny, and that&#8217;s fine<\/h4>\n<p class=\"wp-block-paragraph\">Under RRF with <code>rank_constant=60<\/code>, each sub-query contributes at most <code>1\/61 \u2248 0.0164<\/code> to the final score. Two sub-queries \u2192 max score \u2248 0.033. Confusing the first time you see it. The numbers don&#8217;t mean what BM25 scores mean; only the ordering does.<\/p>\n<h4 class=\"wp-block-heading\">Tika MIME detection has gaps<\/h4>\n<p class=\"wp-block-paragraph\"><code>.srt<\/code>, <code>.eml<\/code>, and <code>.epub<\/code> all &#8220;should work&#8221; with stock Tika 3.x and <em>do<\/em> work, but sometimes Tika misdetects a corrupt file or a non-standard extension. The fix is a <code>tika-mimetypes.xml<\/code> override; it&#8217;s worth keeping one in the FSCrawler config volume from the start.<\/p>\n<h4 class=\"wp-block-heading\">OCR quality on screenshots is bad<\/h4>\n<p class=\"wp-block-paragraph\"><code>pdf_strategy: ocr_and_text<\/code> works great on scanned book pages. On screenshots of UIs it produces 10-15 characters of noise like <code>\"= zs Bs ARK\"<\/code>. This is not configurable away. It&#8217;s an OCR-quality ceiling. Accept it as the baseline. If you need better OCR on UI screenshots, you&#8217;re looking at a separate vision-LLM pass, not a Tika tweak.<\/p>\n<h4 class=\"wp-block-heading\">Build the monitoring before the demo<\/h4>\n<p class=\"wp-block-paragraph\">The day a NAS mount dies silently, you want the page to fire, not for a user to notice retrieval has degraded. Page-level alerts on: cluster status, model state, snapshot age. Warn-level on: embedding ratio drift, NFS mount status, FSCrawler exit code.<\/p>\n<h4 class=\"wp-block-heading\">Run the embed inside the cluster, not as a separate service<\/h4>\n<p class=\"wp-block-paragraph\">The ML-Commons plugin lets the embedding model live in OpenSearch. Query-time embedding is one query path, one network hop. Tried briefly to run embeddings through LiteLLM (out-of-cluster); the extra hop adds 10-30 ms per query and you have to manage another service&#8217;s uptime. Keep it inside.<\/p>\n<hr class=\"wp-block-separator\" \/>\n<h3 class=\"wp-block-heading\">4. What&#8217;s next on the roadmap<\/h3>\n<ul class=\"wp-block-list\">\n<li>Move the whole stack onto a Proxmox VM so the GPU box can go back to being a GPU box (see <code>MIGRATION_PLAN_PROXMOX_VM.md<\/code>).<\/li>\n<li>Switch the answerer LLM from a generic chat completion to one fine-tuned for citation-style RAG.<\/li>\n<\/ul>\n<h4 class=\"wp-block-heading\">Recently shipped<\/h4>\n<ul class=\"wp-block-list\">\n<li><strong>Source-isolated search routes (2026-05-28).<\/strong> <code>portable_docs<\/code> and <code>airflow_docs<\/code> are now physically separated indices. <code>search-api<\/code> exposes <code>POST \/search\/portable<\/code> and <code>POST \/search\/airflow<\/code>, each hard-wired to its index so callers cannot leak documents across sources. Full deploy steps in <code>deployment\/SEPARATION_DEPLOY_2026-05-28.md<\/code>.<\/li>\n<li><strong><code>.vtt<\/code> subtitle support (2026-05-28).<\/strong> A <code>tika-config.xml<\/code> override mounted into the FSCrawler container maps <code>text\/vtt<\/code> to the plaintext parser, so the 55,000 <code>.vtt<\/code> files on the NAS now flow through <code>nlp-pipeline<\/code> like any other text source.<\/li>\n<li><strong><code>sync_to_live<\/code> wrapper.<\/strong> <code>deployment\/scripts\/sync_to_live.sh<\/code> rsyncs the git checkout (<code>\/home\/llm\/gitrepos\/...\/big-search\/<\/code>) into the live deploy dir (<code>\/home\/llm\/knowledge-graphs\/big-search\/<\/code>) that container bind-mounts read from. Two paths exist because the live dir is root-owned and holds FSCrawler runtime files (<code>_status.json<\/code>, <code>_meta.json<\/code>) that mustn&#8217;t be clobbered, the wrapper excludes those, runs dry-run by default, and prints the <code>docker compose up -d --force-recreate<\/code> follow-up.<\/li>\n<\/ul>\n<hr class=\"wp-block-separator\" \/>\n<h3 class=\"wp-block-heading\">5. References from the build<\/h3>\n<ul class=\"wp-block-list\">\n<li><code>AI_SEARCH_BLUEPRINT.md<\/code>: original target architecture.<\/li>\n<li><code>AI_SEARCH_IMPLEMENTATION_LOG.md<\/code>: chronological change history with every gotcha.<\/li>\n<li><code>INCIDENT_REPORT_INDEX_DATA_LOSS.md<\/code>: the 276k-doc-loss writeup.<\/li>\n<li><code>REPAIR_PLAN_2026-05-17.md<\/code>: post-upgrade audit and the phased recovery, including the NMSLIB \u2192 Faiss migration.<\/li>\n<li><code>search-api\/README.md<\/code>: query patterns and raw OpenSearch equivalents.<\/li>\n<li><code>deployment\/MONITORING_RECOMMENDATIONS.md<\/code>: the monitoring design before it was built.<\/li>\n<\/ul>\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h2 class=\"wp-block-heading\">Get the next one in your inbox<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Enjoyed this? I write about local AI, self-hosting, and cloud cost ninjutsu. Join the Ninja Dispatch for occasional, high-signal notes. No fluff.<\/p>\n\n\n\n\t\t\t<form id=\"sib_signup_form_2\" method=\"post\" class=\"sib_signup_form\">\n\t\t\t\t<div class=\"sib_loader\" style=\"display:none;\"><img\n\t\t\t\t\t\t\tsrc=\"https:\/\/yyc.ninja\/wp-includes\/images\/spinner.gif\" alt=\"loader\"><\/div>\n\t\t\t\t<input type=\"hidden\" name=\"sib_form_action\" value=\"subscribe_form_submit\">\n\t\t\t\t<input type=\"hidden\" name=\"sib_form_id\" value=\"2\">\n                <input type=\"hidden\" name=\"sib_form_alert_notice\" value=\"Please fill out all required fields.\">\n                <input type=\"hidden\" name=\"sib_form_invalid_email_notice\" value=\"Please enter a valid email address.\">\n                <input type=\"hidden\" name=\"sib_security\" value=\"9505f57aea\">\n\t\t\t\t<div class=\"sib_signup_box_inside_2\">\n\t\t\t\t\t<div style=\"\/*display:none*\/\" class=\"sib_msg_disp\">\n\t\t\t\t\t<\/div>\n                    \t\t\t\t\t<p class=\"sib-email-area\">\n    <input type=\"email\" class=\"sib-email-area\" name=\"email\" placeholder=\"you@example.com\" required=\"required\">\n<\/p>\n<p class=\"sib-FIRSTNAME-area\">\n    <input type=\"text\" class=\"sib-FIRSTNAME-area\" name=\"FIRSTNAME\" placeholder=\"First name (optional)\">\n<\/p>\n<input type=\"hidden\" class=\"sib-SOURCE_DOMAIN-area\" name=\"SOURCE_DOMAIN\" value=\"yyc.ninja\">\n<input type=\"hidden\" class=\"sib-SIGNUP_SOURCE-area\" name=\"SIGNUP_SOURCE\" value=\"post\">\n<p>\n    <input type=\"submit\" class=\"sib-default-btn\" value=\"Join the Ninja Dispatch\">\n<\/p>\n<p class=\"ninja-fineprint\">No spam. Unsubscribe anytime.<\/p>\t\t\t\t<\/div>\n\t\t\t<\/form>\n\t\t\t<style>\n\t\t\t\tform#sib_signup_form_2{background:#192a3d;border-radius:10px;padding:16px;font-family:inherit}form#sib_signup_form_2 input[type=text],form#sib_signup_form_2 input[type=email],form#sib_signup_form_2 select{width:100%;box-sizing:border-box;margin:0 0 10px 0;padding:10px 12px;border:1px solid #33465c;border-radius:8px;background:#0f1c29;color:#e8eef5;font-size:14px}form#sib_signup_form_2 input::placeholder{color:#8a9bb0}form#sib_signup_form_2 .sib-default-btn{width:100%;margin:2px 0 0 0;padding:11px 14px;background:#e63946;color:#ffffff;border:0;border-radius:8px;font-size:15px;font-weight:600;cursor:pointer}form#sib_signup_form_2 .sib-default-btn:hover{background:#d02f3c}form#sib_signup_form_2 p{margin:0 0 10px 0}form#sib_signup_form_2 .ninja-fineprint{margin:8px 0 0;font-size:11px;color:#8a9bb0;text-align:center}form#sib_signup_form_2 p.sib-alert-message {\n    padding: 6px 12px;\n    margin-bottom: 20px;\n    border: 1px solid transparent;\n    border-radius: 4px;\n    -webkit-box-sizing: border-box;\n    -moz-box-sizing: border-box;\n    box-sizing: border-box;\n}\nform#sib_signup_form_2 p.sib-alert-message-error {\n    background-color: #f2dede;\n    border-color: #ebccd1;\n    color: #a94442;\n}\nform#sib_signup_form_2 p.sib-alert-message-success {\n    background-color: #dff0d8;\n    border-color: #d6e9c6;\n    color: #3c763d;\n}\nform#sib_signup_form_2 p.sib-alert-message-warning {\n    background-color: #fcf8e3;\n    border-color: #faebcc;\n    color: #8a6d3b;\n}\n\t\t\t<\/style>\n\t\t\t\n","protected":false},"excerpt":{"rendered":"<p>This is the build log for a working system as of 2026-05-28. ~32,000 documents indexed across personal documents, course materials, code, and a PDF library. Hybrid BM25 + k-NN semantic retrieval with Reciprocal Rank Fusion, OCR for images and scanned PDFs, and a chat-style Q&#038;A agent on top.<\/p>\n","protected":false},"author":1,"featured_media":3475,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[12,19],"tags":[43,44,42,39,38,41,40],"class_list":["post-3423","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-how-to","category-tutorial","tag-ai","tag-data-engineering","tag-fscrawler","tag-hybrid-search","tag-opensearch","tag-rag","tag-self-hosted"],"featured_image_urls_v2":{"full":["https:\/\/yyc.ninja\/wp-content\/uploads\/2026\/10\/how-to-build-big-search-a-self-hosted-hybrid-ai-search-platf-cover-logo.png",1200,630,false],"thumbnail":["https:\/\/yyc.ninja\/wp-content\/uploads\/2026\/10\/how-to-build-big-search-a-self-hosted-hybrid-ai-search-platf-cover-logo-150x150.png",150,150,true],"medium":["https:\/\/yyc.ninja\/wp-content\/uploads\/2026\/10\/how-to-build-big-search-a-self-hosted-hybrid-ai-search-platf-cover-logo-300x158.png",300,158,true],"medium_large":["https:\/\/yyc.ninja\/wp-content\/uploads\/2026\/10\/how-to-build-big-search-a-self-hosted-hybrid-ai-search-platf-cover-logo-768x403.png",768,403,true],"large":["https:\/\/yyc.ninja\/wp-content\/uploads\/2026\/10\/how-to-build-big-search-a-self-hosted-hybrid-ai-search-platf-cover-logo-1024x538.png",1024,538,true],"RoboGalleryMansoryImagesCenter":["https:\/\/yyc.ninja\/wp-content\/uploads\/2026\/10\/how-to-build-big-search-a-self-hosted-hybrid-ai-search-platf-cover-logo-600x630.png",600,630,true],"RoboGalleryPreload":["https:\/\/yyc.ninja\/wp-content\/uploads\/2026\/10\/how-to-build-big-search-a-self-hosted-hybrid-ai-search-platf-cover-logo-100x53.png",100,53,true],"1536x1536":["https:\/\/yyc.ninja\/wp-content\/uploads\/2026\/10\/how-to-build-big-search-a-self-hosted-hybrid-ai-search-platf-cover-logo.png",1200,630,false],"2048x2048":["https:\/\/yyc.ninja\/wp-content\/uploads\/2026\/10\/how-to-build-big-search-a-self-hosted-hybrid-ai-search-platf-cover-logo.png",1200,630,false]},"post_excerpt_stackable_v2":"<p>This is the build log for a working system as of 2026-05-28. ~32,000 documents indexed across personal documents, course materials, code, and a PDF library. Hybrid BM25 + k-NN semantic retrieval with Reciprocal Rank Fusion, OCR for images and scanned PDFs, and a chat-style Q&#038;A agent on top.<\/p>\n","category_list_v2":"<a href=\"https:\/\/yyc.ninja\/index.php\/category\/how-to\/\" rel=\"category tag\">How To Ninja<\/a>, <a href=\"https:\/\/yyc.ninja\/index.php\/category\/how-to\/tutorial\/\" rel=\"category tag\">Tutorial<\/a>","author_info_v2":{"name":"YYC Ninja","url":"https:\/\/yyc.ninja\/index.php\/author\/admin\/"},"comments_num_v2":"0 comments","_links":{"self":[{"href":"https:\/\/yyc.ninja\/index.php\/wp-json\/wp\/v2\/posts\/3423","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/yyc.ninja\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/yyc.ninja\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/yyc.ninja\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/yyc.ninja\/index.php\/wp-json\/wp\/v2\/comments?post=3423"}],"version-history":[{"count":4,"href":"https:\/\/yyc.ninja\/index.php\/wp-json\/wp\/v2\/posts\/3423\/revisions"}],"predecessor-version":[{"id":3502,"href":"https:\/\/yyc.ninja\/index.php\/wp-json\/wp\/v2\/posts\/3423\/revisions\/3502"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/yyc.ninja\/index.php\/wp-json\/wp\/v2\/media\/3475"}],"wp:attachment":[{"href":"https:\/\/yyc.ninja\/index.php\/wp-json\/wp\/v2\/media?parent=3423"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/yyc.ninja\/index.php\/wp-json\/wp\/v2\/categories?post=3423"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/yyc.ninja\/index.php\/wp-json\/wp\/v2\/tags?post=3423"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}