Hybrid Search

WitWiki implements hybrid search combining full-text search (FTS) with vector similarity search for relevant results across the knowledge base.

Checked against the code 2026-09-06.

Overview

The hybrid search approach provides:

• Keyword search: Exact word matching using PostgreSQL tsvector
• Semantic search: Conceptual similarity using vector embeddings (pgvector)
• Balanced results: Combines both approaches with weighted scoring

Architecture

┌─────────────────────────────────────────────────────────┐
│                    Search Query                          │
│                  (e.g., "authentication")                 │
└────────────────────┬────────────────────────────────────┘
                     │
         ┌───────────┴───────────┐
         │                       │
┌────────▼──────────┐  ┌─────────▼──────────┐
│   FTS Search      │  │   Vector Search    │
│   (tsvector)      │  │   (pgvector)       │
│   - Exact words   │  │   - Concepts       │
│   - Exact matches │  │   - Semantic       │
└────────┬──────────┘  └─────────┬──────────┘
         │                       │
         └───────────┬───────────┘
                     │
         ┌───────────▼───────────┐
         │   Combine & Rank      │
         │   (weighted scoring)  │
         └───────────┬───────────┘
                     │
         ┌───────────▼───────────┐
         │   Search Results      │
         │   (sorted by relevance)│
         └────────────────────────┘

The two components live in api/internal/search (FTS + vector combination, weighted scoring) and api/internal/embedding (model selection, embedding generation). The DefaultSearcher runs FTS first, optionally runs vector search when enabled, then merges and ranks the results.

Models

Prototype: all-MiniLM-L6-v2

• Size: ~60MB
• Dimensions: 384
• Use: Local development, testing, prototyping (CPU-friendly)
• This is the default (embedding.DefaultModelName) and matches the embedding vector(384) column.

Production: all-mpnet-base-v2

• Size: ~1.5GB
• Dimensions: 768
• Use: Higher-quality production embeddings

Select the model via the EMBEDDING_MODEL environment variable. The model path can be overridden with EMBEDDING_MODEL_PATH.

Embedding generation

Embeddings are generated on page ingest when EMBEDDING_GENERATE_ON_INGEST=true (see embedding.Config.GenerateOnIngest). This behavior was added post-2026-04-28: rather than backfilling embeddings in a separate pass, a page's vector is produced as part of writing/ingesting the page, so search stays current with edits. Each row also records embedding_model_version and embedding_generated_at for tracking.

# Local development (prototype model, generate vectors on ingest)
EMBEDDING_MODEL=all-MiniLM-L6-v2
EMBEDDING_GENERATE_ON_INGEST=true

# Production (higher-quality model)
EMBEDDING_MODEL=all-mpnet-base-v2

API

GET /api/search

Perform a search query against the current org/project's wiki pages.

Parameters:

Response:

{
  "pages": [
    {
      "path": "concepts/oauth2",
      "title": "OAuth2 Authorization"
    }
  ]
}

The HTTP endpoint is the thin entry point; the hybrid FTS + vector combination and weighted scoring happen inside the search package. Weights and the vector similarity threshold are configured via search.Config (VectorSearchWeight, FTSWeight, VectorSearchThreshold).

Database schema

Migration 0017_wiki_embedding.sql adds the hybrid-search columns and indexes to wiki_pages:

• embedding vector(384) — vector embedding for semantic search (all-MiniLM-L6-v2)
• embedding_model_version / embedding_generated_at — embedding metadata
• search_text tsvector — keyword component, backed by a GIN index
• An HNSW index (wiki_pages_embedding_idx) on the embedding column using vector_cosine_ops, created only when pgvector exposes the operator class so the server can still boot and serve searches via sequential scan until pgvector is upgraded (HNSW requires pgvector >= 0.5.0).

The FTS search index work landed separately in 0018_wiki_search_index.sql.

# Apply via the migration runner (migrations are embedded and run on startup);
# pgvector must be available in the Postgres instance first.

HNSW index performance

The HNSW index accelerates approximate nearest-neighbor lookups for the vector component. It is created conditionally (see above) so a Postgres instance without a new-enough pgvector still functions — vector queries simply fall back to a sequential scan until the extension is upgraded and the index is created.

Testing

go test -v ./internal/embedding/...
go test -v ./internal/search/...

Related

• search — Search overview (FTS + semantic) from the product side

Future enhancements

• Model auto-downloading on first use
• Embedding cache for repeated queries
• Expose mode (fts / vector / hybrid) and prefix filtering on the HTTP API
• Vector search analytics dashboard