How to build

Audience: Someone with a NAS full of documents, a couple of Linux boxes, and an itch to make “search my stuff” actually work, including semantic search, OCR, and an LLM-powered Q&A layer.

This is the build log for a working system as of 2026-05-28. ~32,000 documents indexed across personal documents, course materials, code, and a PDF library. Hybrid BM25 + k-NN semantic retrieval with Reciprocal Rank Fusion, OCR for images and scanned PDFs, and a chat-style Q&A agent on top.


0. What you end up with

You ──► Dify chat agent
         │
         ▼ POST /search/portable  (or /search/airflow)
       search-api  (FastAPI on :8088)
         │
         ▼ hybrid: BM25 + k-NN, fused with RRF
       OpenSearch 2.19.5  (:9200)
         │
         ├─► Faiss HNSW index over content_vector
         ├─► all-MiniLM-L6-v2 embedding model (in-cluster, 384-dim)
         └─► nlp-pipeline (ingest)
         ▲
         │ _bulk
       FSCrawler  ×N  (one per source)
         │
         ▼ reads files
       NAS via NFSv4

Plus monitoring (Prometheus + Grafana + Loki), nightly snapshot backups to the NAS, and an LLM gateway (LiteLLM) for unified embedding/completion access.


1. The pieces and why each one

1.1 OpenSearch (not Elasticsearch)

Apache-2.0 licensed fork of Elasticsearch 7.10. We picked it for two specific reasons:

  • neural_search plugin ships out of the box. No vector DB to deploy separately. Embeddings live next to the BM25 index.
  • hybrid query with score-ranker-processor (added in OS 2.17) supports true Reciprocal Rank Fusion. Hybrid keyword + semantic is one query, not two.

Started on 2.11, upgraded to 2.19.5 mid-project for RRF and inline k-NN filters. Lesson: pick 2.19+ from day one if you can.

1.2 FSCrawler

Crawls a filesystem, runs Apache Tika on each file, and _bulk-writes the extracted text into OpenSearch. Handles:

  • Text formats (Markdown, code, JSON, YAML, HTML, plain text).
  • Office formats (Word, Excel, PowerPoint) via Tika.
  • PDFs with two strategies: text (fast, only extracts already-embedded text) and ocr_and_text (runs Tesseract on image-only pages). We use ocr_and_text.
  • Images (.png, .jpg, .webp, etc.): Tika invokes Tesseract for OCR.
  • Subtitles (.srt, .vtt): treated as plain text once Tika MIME detection is configured.

One FSCrawler container per data source so each can have its own scan schedule and excludes list. We run one for /portable/, one for /airflow/ on the NAS. Each writes to its own index.

1.3 Apache Tika (inside FSCrawler)

Format detector + text extractor. The thing that makes a PDF and a .docx and a .png all look the same to the indexer downstream. Tika’s MIME detection is almost good enough out of the box. The gaps we hit:

  • .srt was being parsed but not always classified as text, fixed with an explicit MIME override.
  • Image-only PDFs need OCR enabled (pdf_strategy: ocr_and_text).
  • Standalone images need ocr.enabled: true (default is on, but worth verifying).

1.4 The embedding model: all-MiniLM-L6-v2

384-dimensional sentence-transformer, runs on CPU inside OpenSearch via the _plugins/_ml API. Picked because:

  • Cheap to run (~5 ms per query on a laptop CPU).
  • Good enough quality for retrieval-grade similarity.
  • Fits in memory next to the search workload.

Downside: 384-dim is small. If quality plateaus, swap to nomic-embed-text (768-dim, also self-hostable on Ollama), we already have it wired into LiteLLM for the next iteration.

1.5 Faiss + HNSW (the k-NN engine)

OpenSearch ships three k-NN engines: Lucene, NMSLIB, Faiss. We started on NMSLIB (the OS 2.11 default), hit a wall on filtered k-NN queries (NMSLIB doesn’t support them), and migrated all three indices to Faiss. With Faiss:

  • The filter clause on a neural sub-query is pushed into the HNSW graph traversal.
  • A query for “Markdown files about Kubernetes” returns documents that are both semantically about Kubernetes and actually .md, no client-side post-pruning.

Trade-off: Faiss requires reindexing the whole field. Plan for it from day one.

1.6 search-api (FastAPI)

Thin Python service that:

  • Accepts {query, top_k, filters} from clients.
  • Builds the OpenSearch hybrid query JSON (one match clause + one neural clause, fused by the RRF pipeline).
  • Returns the top-k hits with the document content and metadata.

Why a service in front of OpenSearch instead of letting Dify hit OpenSearch directly: it normalizes filter syntax (callers can send filters as an object, a JSON string, or a Python repr, all three coerce correctly), and it lets us swap the underlying engine without touching every client.

1.7 Dify (the chat agent layer)

Open-source LLM application platform. Each chat flow is a graph of nodes: router LLM → HTTP tool (call search-api) → answerer LLM. We use it instead of writing custom RAG code because the visual editor makes it trivial to change the prompt or swap the LLM. The agent flow that’s live calls POST /search with filters chosen by the router LLM, then synthesizes an answer with citations to path.virtual.

1.8 LiteLLM (the LLM gateway)

OpenAI-compatible proxy in front of every model we use: Ollama (local), Anthropic (cloud), OpenAI (cloud), Nomic embedding. Means every code path that wants an LLM speaks one dialect, and we get unified request logging + cost tracking for free.

1.9 Monitoring (the unglamorous foundation)

  • Prometheus scrapes:
  • opensearch-exporter (cluster, JVM, index metrics)
  • bigsearch_metrics: a custom Python sidecar that emits 11 domain signals (bigsearch_ml_model_deployed, bigsearch_embed_latency_seconds, bigsearch_index_with_vector_ratio, bigsearch_snapshot_age_seconds, bigsearch_nas_reachable, etc.)
  • nas-mount-monitor: emits 0/1 if NFS mounts are alive
  • Grafana dashboard bigsearch-overview shows it all.
  • Loki + Alloy ship container logs.
  • 13 alert rules cover: cluster red, model not deployed, NAS unreachable, snapshot too old, embedding-ratio drift, search-api down.

You learn the value of monitoring the day FSCrawler silently deletes 276,000 documents because an NFS mount unmounted itself. Build the alerts first.


2. The build, in order

Step 1: bring up OpenSearch

docker compose up -d opensearch-node1 opensearch-dashboards. Single node, security plugin disabled (it’s on a LAN). 8 GiB heap to start.

Verify with GET /_cluster/health → green. Open Dashboards on :5601.

Step 2: deploy the ML model

curl -X POST "http://localhost:9200/_plugins/_ml/models/_register" \
  -H 'Content-Type: application/json' \
  -d '{
    "name": "huggingface/sentence-transformers/all-MiniLM-L6-v2",
    "version": "1.0.1",
    "model_format": "TORCH_SCRIPT"
  }'

Then _deploy the returned model_id. The model lives inside OpenSearch from this point on. Verify by predicting a one-sentence embedding (should return 384 floats).

Gotcha we hit: an OS 2.11 → 2.19 upgrade left a stale cpu-precxx11 PyTorch cache whose libstdc++ lacked CXXABI_1.3.9. Model deployment threw UnsatisfiedLinkError. Fix: set PYTORCH_PRECXX11=false env var on the OpenSearch container, clear data/ml_cache/pytorch, recreate, redeploy.

Step 3: create the ingest pipeline

curl -X PUT "http://localhost:9200/_ingest/pipeline/nlp-pipeline" \
  -H 'Content-Type: application/json' \
  -d '{
    "processors": [{
      "text_embedding": {
        "model_id": "BST8XJwBu6n7bcim86zF",
        "field_map": {"content": "content_vector"}
      }
    }]
  }'

Now every document indexed with ?pipeline=nlp-pipeline gets a content_vector field populated by the model.

Step 4: create the index mapping (Faiss from day one)

The critical bit:

{
  "mappings": {
    "properties": {
      "content": {"type": "text"},
      "content_vector": {
        "type": "knn_vector",
        "dimension": 384,
        "method": {
          "name": "hnsw",
          "engine": "faiss",
          "space_type": "l2",
          "parameters": {"ef_construction": 256, "m": 16}
        }
      }
    }
  },
  "settings": {
    "index.knn": true,
    "default_pipeline": "nlp-pipeline"
  }
}

engine: faiss is what enables filtered k-NN later. Don’t skip it.

Step 5: the hybrid search pipeline

curl -X PUT "http://localhost:9200/_search/pipeline/hybrid-rrf-pipeline" \
  -H 'Content-Type: application/json' \
  -d '{
    "phase_results_processors": [{
      "score-ranker-processor": {
        "combination": {"technique": "rrf", "rank_constant": 60}
      }
    }]
  }'

This is how BM25 and k-NN get fused into one ranked list.

Step 6: bring up FSCrawler

_settings.json per crawler. Key fields:

{
  "fs": {
    "url": "/tmp/es/portable",
    "update_rate": "15m",
    "remove_deleted": false,
    "continue_on_error": true,
    "ocr": {"enabled": true, "pdf_strategy": "ocr_and_text", "language": "eng"},
    "excludes": ["*/node_modules/*", "*/.git/*", "*/__pycache__/*", ...]
  },
  "elasticsearch": {
    "urls": ["http://opensearch-node1:9200"],
    "index": "portable_docs",
    "pipeline": "nlp-pipeline"
  }
}

remove_deleted: false is the most important line in the entire system. The default (true) means FSCrawler will delete documents from OpenSearch if it can’t find the source file. If your NFS mount drops, you lose your index. We learned this the hard way (incident report linked at the end).

Step 7: search-api

FastAPI service. ~150 lines. Builds and proxies the hybrid query, normalizes filter inputs. The whole point is to have one endpoint that the Dify agent (and any other consumer) can hit.

Step 8: Dify

Wire a custom HTTP tool against POST /search. Build a flow: Start → Router LLM (pick doc_type filter) → HTTP tool → Answerer LLM (cite path.virtual) → End. Single-shot (no re-search loop) but enough to validate the chat → retrieval → LLM loop end-to-end.

Step 9: monitoring and snapshots

  • Set up the snapshot repo pointing at a NAS path. Run a daily cron that calls _snapshot/.../snapshot_$(date +%Y%m%d)?wait_for_completion=true.
  • Deploy the Prometheus exporter sidecar.
  • Write bigsearch_metrics.py to emit the things the off-the-shelf exporter doesn’t: ML model state, embedding coverage ratio, NFS mount status.
  • Load the Grafana dashboard JSON.
  • Configure 5-10 alert rules. Start with: cluster red, model not deployed, snapshot age > 36 h, embedding ratio drops > 5%, NFS mount missing.

3. Lessons that cost real time

remove_deleted: true is a footgun

When NFS mounts dropped, FSCrawler walked the (now-empty) tree, decided 276,000 documents had been “deleted,” and removed them all from the index. The fix is one line. Set it before you have a corpus to lose.

NMSLIB → Faiss is non-negotiable

Inline k-NN filters are the difference between “semantic search that returns relevant results” and “semantic search that returns whatever its closest neighbours are, ignoring your filters.” Pick Faiss at index-creation time. Reindexing 30,000 docs later isn’t catastrophic, but it’s hours you don’t need to spend.

Hybrid scores are tiny, and that’s fine

Under RRF with rank_constant=60, each sub-query contributes at most 1/61 ≈ 0.0164 to the final score. Two sub-queries → max score ≈ 0.033. Confusing the first time you see it. The numbers don’t mean what BM25 scores mean; only the ordering does.

Tika MIME detection has gaps

.srt, .eml, and .epub all “should work” with stock Tika 3.x and do work, but sometimes Tika misdetects a corrupt file or a non-standard extension. The fix is a tika-mimetypes.xml override; it’s worth keeping one in the FSCrawler config volume from the start.

OCR quality on screenshots is bad

pdf_strategy: ocr_and_text works great on scanned book pages. On screenshots of UIs it produces 10-15 characters of noise like "= zs Bs ARK". This is not configurable away. It’s an OCR-quality ceiling. Accept it as the baseline. If you need better OCR on UI screenshots, you’re looking at a separate vision-LLM pass, not a Tika tweak.

Build the monitoring before the demo

The day a NAS mount dies silently, you want the page to fire, not for a user to notice retrieval has degraded. Page-level alerts on: cluster status, model state, snapshot age. Warn-level on: embedding ratio drift, NFS mount status, FSCrawler exit code.

Run the embed inside the cluster, not as a separate service

The ML-Commons plugin lets the embedding model live in OpenSearch. Query-time embedding is one query path, one network hop. Tried briefly to run embeddings through LiteLLM (out-of-cluster); the extra hop adds 10-30 ms per query and you have to manage another service’s uptime. Keep it inside.


4. What’s next on the roadmap

  • Move the whole stack onto a Proxmox VM so the GPU box can go back to being a GPU box (see MIGRATION_PLAN_PROXMOX_VM.md).
  • Switch the answerer LLM from a generic chat completion to one fine-tuned for citation-style RAG.

Recently shipped

  • Source-isolated search routes (2026-05-28). portable_docs and airflow_docs are now physically separated indices. search-api exposes POST /search/portable and POST /search/airflow, each hard-wired to its index so callers cannot leak documents across sources. Full deploy steps in deployment/SEPARATION_DEPLOY_2026-05-28.md.
  • .vtt subtitle support (2026-05-28). A tika-config.xml override mounted into the FSCrawler container maps text/vtt to the plaintext parser, so the 55,000 .vtt files on the NAS now flow through nlp-pipeline like any other text source.
  • sync_to_live wrapper. deployment/scripts/sync_to_live.sh rsyncs the git checkout (/home/llm/gitrepos/.../big-search/) into the live deploy dir (/home/llm/knowledge-graphs/big-search/) that container bind-mounts read from. Two paths exist because the live dir is root-owned and holds FSCrawler runtime files (_status.json, _meta.json) that mustn’t be clobbered, the wrapper excludes those, runs dry-run by default, and prints the docker compose up -d --force-recreate follow-up.

5. References from the build

  • AI_SEARCH_BLUEPRINT.md: original target architecture.
  • AI_SEARCH_IMPLEMENTATION_LOG.md: chronological change history with every gotcha.
  • INCIDENT_REPORT_INDEX_DATA_LOSS.md: the 276k-doc-loss writeup.
  • REPAIR_PLAN_2026-05-17.md: post-upgrade audit and the phased recovery, including the NMSLIB → Faiss migration.
  • search-api/README.md: query patterns and raw OpenSearch equivalents.
  • deployment/MONITORING_RECOMMENDATIONS.md: the monitoring design before it was built.

Get the next one in your inbox

Enjoyed this? I write about local AI, self-hosting, and cloud cost ninjutsu. Join the Ninja Dispatch for occasional, high-signal notes. No fluff.