
Audience: Someone with a NAS full of documents, a couple of Linux boxes, and an itch to make “search my stuff” actually work, including semantic search, OCR, and an LLM-powered Q&A layer.
This is the build log for a working system as of 2026-05-28. ~32,000 documents indexed across personal documents, course materials, code, and a PDF library. Hybrid BM25 + k-NN semantic retrieval with Reciprocal Rank Fusion, OCR for images and scanned PDFs, and a chat-style Q&A agent on top.
0. What you end up with
You ──► Dify chat agent
│
▼ POST /search/portable (or /search/airflow)
search-api (FastAPI on :8088)
│
▼ hybrid: BM25 + k-NN, fused with RRF
OpenSearch 2.19.5 (:9200)
│
├─► Faiss HNSW index over content_vector
├─► all-MiniLM-L6-v2 embedding model (in-cluster, 384-dim)
└─► nlp-pipeline (ingest)
▲
│ _bulk
FSCrawler ×N (one per source)
│
▼ reads files
NAS via NFSv4
Plus monitoring (Prometheus + Grafana + Loki), nightly snapshot backups to the NAS, and an LLM gateway (LiteLLM) for unified embedding/completion access.
1. The pieces and why each one
1.1 OpenSearch (not Elasticsearch)
Apache-2.0 licensed fork of Elasticsearch 7.10. We picked it for two specific reasons:
neural_searchplugin ships out of the box. No vector DB to deploy separately. Embeddings live next to the BM25 index.hybridquery withscore-ranker-processor(added in OS 2.17) supports true Reciprocal Rank Fusion. Hybrid keyword + semantic is one query, not two.
Started on 2.11, upgraded to 2.19.5 mid-project for RRF and inline k-NN filters. Lesson: pick 2.19+ from day one if you can.
1.2 FSCrawler
Crawls a filesystem, runs Apache Tika on each file, and _bulk-writes the extracted text into OpenSearch. Handles:
- Text formats (Markdown, code, JSON, YAML, HTML, plain text).
- Office formats (Word, Excel, PowerPoint) via Tika.
- PDFs with two strategies:
text(fast, only extracts already-embedded text) andocr_and_text(runs Tesseract on image-only pages). We useocr_and_text. - Images (
.png,.jpg,.webp, etc.): Tika invokes Tesseract for OCR. - Subtitles (
.srt,.vtt): treated as plain text once Tika MIME detection is configured.
One FSCrawler container per data source so each can have its own scan schedule and excludes list. We run one for /portable/, one for /airflow/ on the NAS. Each writes to its own index.
1.3 Apache Tika (inside FSCrawler)
Format detector + text extractor. The thing that makes a PDF and a .docx and a .png all look the same to the indexer downstream. Tika’s MIME detection is almost good enough out of the box. The gaps we hit:
.srtwas being parsed but not always classified as text, fixed with an explicit MIME override.- Image-only PDFs need OCR enabled (
pdf_strategy: ocr_and_text). - Standalone images need
ocr.enabled: true(default is on, but worth verifying).
1.4 The embedding model: all-MiniLM-L6-v2
384-dimensional sentence-transformer, runs on CPU inside OpenSearch via the _plugins/_ml API. Picked because:
- Cheap to run (~5 ms per query on a laptop CPU).
- Good enough quality for retrieval-grade similarity.
- Fits in memory next to the search workload.
Downside: 384-dim is small. If quality plateaus, swap to nomic-embed-text (768-dim, also self-hostable on Ollama), we already have it wired into LiteLLM for the next iteration.
1.5 Faiss + HNSW (the k-NN engine)
OpenSearch ships three k-NN engines: Lucene, NMSLIB, Faiss. We started on NMSLIB (the OS 2.11 default), hit a wall on filtered k-NN queries (NMSLIB doesn’t support them), and migrated all three indices to Faiss. With Faiss:
- The
filterclause on aneuralsub-query is pushed into the HNSW graph traversal. - A query for “Markdown files about Kubernetes” returns documents that are both semantically about Kubernetes and actually
.md, no client-side post-pruning.
Trade-off: Faiss requires reindexing the whole field. Plan for it from day one.
1.6 search-api (FastAPI)
Thin Python service that:
- Accepts
{query, top_k, filters}from clients. - Builds the OpenSearch hybrid query JSON (one
matchclause + oneneuralclause, fused by the RRF pipeline). - Returns the top-k hits with the document content and metadata.
Why a service in front of OpenSearch instead of letting Dify hit OpenSearch directly: it normalizes filter syntax (callers can send filters as an object, a JSON string, or a Python repr, all three coerce correctly), and it lets us swap the underlying engine without touching every client.
1.7 Dify (the chat agent layer)
Open-source LLM application platform. Each chat flow is a graph of nodes: router LLM → HTTP tool (call search-api) → answerer LLM. We use it instead of writing custom RAG code because the visual editor makes it trivial to change the prompt or swap the LLM. The agent flow that’s live calls POST /search with filters chosen by the router LLM, then synthesizes an answer with citations to path.virtual.
1.8 LiteLLM (the LLM gateway)
OpenAI-compatible proxy in front of every model we use: Ollama (local), Anthropic (cloud), OpenAI (cloud), Nomic embedding. Means every code path that wants an LLM speaks one dialect, and we get unified request logging + cost tracking for free.
1.9 Monitoring (the unglamorous foundation)
- Prometheus scrapes:
opensearch-exporter(cluster, JVM, index metrics)bigsearch_metrics: a custom Python sidecar that emits 11 domain signals (bigsearch_ml_model_deployed,bigsearch_embed_latency_seconds,bigsearch_index_with_vector_ratio,bigsearch_snapshot_age_seconds,bigsearch_nas_reachable, etc.)nas-mount-monitor: emits 0/1 if NFS mounts are alive- Grafana dashboard
bigsearch-overviewshows it all. - Loki + Alloy ship container logs.
- 13 alert rules cover: cluster red, model not deployed, NAS unreachable, snapshot too old, embedding-ratio drift, search-api down.
You learn the value of monitoring the day FSCrawler silently deletes 276,000 documents because an NFS mount unmounted itself. Build the alerts first.
2. The build, in order
Step 1: bring up OpenSearch
docker compose up -d opensearch-node1 opensearch-dashboards. Single node, security plugin disabled (it’s on a LAN). 8 GiB heap to start.
Verify with GET /_cluster/health → green. Open Dashboards on :5601.
Step 2: deploy the ML model
curl -X POST "http://localhost:9200/_plugins/_ml/models/_register" \
-H 'Content-Type: application/json' \
-d '{
"name": "huggingface/sentence-transformers/all-MiniLM-L6-v2",
"version": "1.0.1",
"model_format": "TORCH_SCRIPT"
}'
Then _deploy the returned model_id. The model lives inside OpenSearch from this point on. Verify by predicting a one-sentence embedding (should return 384 floats).
Gotcha we hit: an OS 2.11 → 2.19 upgrade left a stale cpu-precxx11 PyTorch cache whose libstdc++ lacked CXXABI_1.3.9. Model deployment threw UnsatisfiedLinkError. Fix: set PYTORCH_PRECXX11=false env var on the OpenSearch container, clear data/ml_cache/pytorch, recreate, redeploy.
Step 3: create the ingest pipeline
curl -X PUT "http://localhost:9200/_ingest/pipeline/nlp-pipeline" \
-H 'Content-Type: application/json' \
-d '{
"processors": [{
"text_embedding": {
"model_id": "BST8XJwBu6n7bcim86zF",
"field_map": {"content": "content_vector"}
}
}]
}'
Now every document indexed with ?pipeline=nlp-pipeline gets a content_vector field populated by the model.
Step 4: create the index mapping (Faiss from day one)
The critical bit:
{
"mappings": {
"properties": {
"content": {"type": "text"},
"content_vector": {
"type": "knn_vector",
"dimension": 384,
"method": {
"name": "hnsw",
"engine": "faiss",
"space_type": "l2",
"parameters": {"ef_construction": 256, "m": 16}
}
}
}
},
"settings": {
"index.knn": true,
"default_pipeline": "nlp-pipeline"
}
}
engine: faiss is what enables filtered k-NN later. Don’t skip it.
Step 5: the hybrid search pipeline
curl -X PUT "http://localhost:9200/_search/pipeline/hybrid-rrf-pipeline" \
-H 'Content-Type: application/json' \
-d '{
"phase_results_processors": [{
"score-ranker-processor": {
"combination": {"technique": "rrf", "rank_constant": 60}
}
}]
}'
This is how BM25 and k-NN get fused into one ranked list.
Step 6: bring up FSCrawler
_settings.json per crawler. Key fields:
{
"fs": {
"url": "/tmp/es/portable",
"update_rate": "15m",
"remove_deleted": false,
"continue_on_error": true,
"ocr": {"enabled": true, "pdf_strategy": "ocr_and_text", "language": "eng"},
"excludes": ["*/node_modules/*", "*/.git/*", "*/__pycache__/*", ...]
},
"elasticsearch": {
"urls": ["http://opensearch-node1:9200"],
"index": "portable_docs",
"pipeline": "nlp-pipeline"
}
}
remove_deleted: false is the most important line in the entire system. The default (true) means FSCrawler will delete documents from OpenSearch if it can’t find the source file. If your NFS mount drops, you lose your index. We learned this the hard way (incident report linked at the end).
Step 7: search-api
FastAPI service. ~150 lines. Builds and proxies the hybrid query, normalizes filter inputs. The whole point is to have one endpoint that the Dify agent (and any other consumer) can hit.
Step 8: Dify
Wire a custom HTTP tool against POST /search. Build a flow: Start → Router LLM (pick doc_type filter) → HTTP tool → Answerer LLM (cite path.virtual) → End. Single-shot (no re-search loop) but enough to validate the chat → retrieval → LLM loop end-to-end.
Step 9: monitoring and snapshots
- Set up the snapshot repo pointing at a NAS path. Run a daily cron that calls
_snapshot/.../snapshot_$(date +%Y%m%d)?wait_for_completion=true. - Deploy the Prometheus exporter sidecar.
- Write
bigsearch_metrics.pyto emit the things the off-the-shelf exporter doesn’t: ML model state, embedding coverage ratio, NFS mount status. - Load the Grafana dashboard JSON.
- Configure 5-10 alert rules. Start with: cluster red, model not deployed, snapshot age > 36 h, embedding ratio drops > 5%, NFS mount missing.
3. Lessons that cost real time
remove_deleted: true is a footgun
When NFS mounts dropped, FSCrawler walked the (now-empty) tree, decided 276,000 documents had been “deleted,” and removed them all from the index. The fix is one line. Set it before you have a corpus to lose.
NMSLIB → Faiss is non-negotiable
Inline k-NN filters are the difference between “semantic search that returns relevant results” and “semantic search that returns whatever its closest neighbours are, ignoring your filters.” Pick Faiss at index-creation time. Reindexing 30,000 docs later isn’t catastrophic, but it’s hours you don’t need to spend.
Hybrid scores are tiny, and that’s fine
Under RRF with rank_constant=60, each sub-query contributes at most 1/61 ≈ 0.0164 to the final score. Two sub-queries → max score ≈ 0.033. Confusing the first time you see it. The numbers don’t mean what BM25 scores mean; only the ordering does.
Tika MIME detection has gaps
.srt, .eml, and .epub all “should work” with stock Tika 3.x and do work, but sometimes Tika misdetects a corrupt file or a non-standard extension. The fix is a tika-mimetypes.xml override; it’s worth keeping one in the FSCrawler config volume from the start.
OCR quality on screenshots is bad
pdf_strategy: ocr_and_text works great on scanned book pages. On screenshots of UIs it produces 10-15 characters of noise like "= zs Bs ARK". This is not configurable away. It’s an OCR-quality ceiling. Accept it as the baseline. If you need better OCR on UI screenshots, you’re looking at a separate vision-LLM pass, not a Tika tweak.
Build the monitoring before the demo
The day a NAS mount dies silently, you want the page to fire, not for a user to notice retrieval has degraded. Page-level alerts on: cluster status, model state, snapshot age. Warn-level on: embedding ratio drift, NFS mount status, FSCrawler exit code.
Run the embed inside the cluster, not as a separate service
The ML-Commons plugin lets the embedding model live in OpenSearch. Query-time embedding is one query path, one network hop. Tried briefly to run embeddings through LiteLLM (out-of-cluster); the extra hop adds 10-30 ms per query and you have to manage another service’s uptime. Keep it inside.
4. What’s next on the roadmap
- Move the whole stack onto a Proxmox VM so the GPU box can go back to being a GPU box (see
MIGRATION_PLAN_PROXMOX_VM.md). - Switch the answerer LLM from a generic chat completion to one fine-tuned for citation-style RAG.
Recently shipped
- Source-isolated search routes (2026-05-28).
portable_docsandairflow_docsare now physically separated indices.search-apiexposesPOST /search/portableandPOST /search/airflow, each hard-wired to its index so callers cannot leak documents across sources. Full deploy steps indeployment/SEPARATION_DEPLOY_2026-05-28.md. .vttsubtitle support (2026-05-28). Atika-config.xmloverride mounted into the FSCrawler container mapstext/vttto the plaintext parser, so the 55,000.vttfiles on the NAS now flow throughnlp-pipelinelike any other text source.sync_to_livewrapper.deployment/scripts/sync_to_live.shrsyncs the git checkout (/home/llm/gitrepos/.../big-search/) into the live deploy dir (/home/llm/knowledge-graphs/big-search/) that container bind-mounts read from. Two paths exist because the live dir is root-owned and holds FSCrawler runtime files (_status.json,_meta.json) that mustn’t be clobbered, the wrapper excludes those, runs dry-run by default, and prints thedocker compose up -d --force-recreatefollow-up.
5. References from the build
AI_SEARCH_BLUEPRINT.md: original target architecture.AI_SEARCH_IMPLEMENTATION_LOG.md: chronological change history with every gotcha.INCIDENT_REPORT_INDEX_DATA_LOSS.md: the 276k-doc-loss writeup.REPAIR_PLAN_2026-05-17.md: post-upgrade audit and the phased recovery, including the NMSLIB → Faiss migration.search-api/README.md: query patterns and raw OpenSearch equivalents.deployment/MONITORING_RECOMMENDATIONS.md: the monitoring design before it was built.
Get the next one in your inbox
Enjoyed this? I write about local AI, self-hosting, and cloud cost ninjutsu. Join the Ninja Dispatch for occasional, high-signal notes. No fluff.

