RAG Pipeline — Retrieval-Augmented Generation

The RAG pipeline is what transforms raiveFlier from a "summarize web search results" tool into a deeply contextual analysis engine. A curated knowledge base of electronic music history — sourced from published books, interviews, longform journalism, and prior analyses — is chunked, embedded, and stored in a ChromaDB vector database. When the system researches an artist or venue, it doesn't just search the web: it retrieves semantically relevant passages from this corpus and injects them into the LLM prompt alongside live API data, producing answers grounded in verified, citation-backed sources.

Ingestion: Source documents (books, articles, interviews, audio transcripts) are processed through format-specific parsers, split into overlapping ~500-token chunks with paragraph-aware boundaries, then enriched with LLM-extracted metadata tags — entity names, genres, geographic references, time periods — before being embedded and stored in ChromaDB.

Retrieval & Natural Language Queries: When a query enters the system, it passes through alias expansion (mapping artist names to known aliases), HyDE-lite query expansion via the LLM, and intelligent filter detection that extracts temporal, geographic, and genre signals from natural language. The expanded query is embedded and searched against the vector store with metadata filters. A tiered retrieval strategy queries books, articles, interviews, and events in parallel, with per-source deduplication (max 3 chunks from any single document) preventing one long source from dominating results.

Result Filtering: Results are filtered at three levels: ChromaDB-native filters for publication dates, source types, and citation tiers; Python post-filters for entity, geographic, and genre tag matching; and a final ranking pass that sorts by cosine similarity with configurable minimum thresholds. The LLM synthesizes the top-ranked passages into natural-language answers with inline citations.

Self-Improving Feedback Loop: Completed flier analyses are automatically ingested back into the corpus as tier-3 sources. The more fliers are analyzed, the richer the context available for future analyses — analyzing a 1993 Orbital flier creates corpus chunks about Orbital's early career, which later enhances analysis of a 1995 flier featuring Orbital alongside Aphex Twin.

Perfecting Natural Language Responses

The RAG pipeline is production-ready and serving queries, but active development continues on three fronts:

• Response Quality Tuning — System prompts are being iteratively refined to produce direct, expert-level answers without preamble or restating the question. Temperature tuning (0.1 for metadata extraction, higher for synthesis) is calibrated per use case.
• Corpus Depth — The knowledge base is being expanded through raiveFeeder with book ingestion, interview transcription, and LLM-guided web crawling that scores page relevance before ingesting content.
• Filter Precision — Genre tag matching currently uses bidirectional substring matching as a post-filter (ChromaDB doesn't natively support $contains on metadata). Work continues on more granular multi-faceted filtering and relevance threshold optimization to reduce noise in results.
• Context Window Budgeting — Each chunk carries a token count for precise context window management, enabling smarter selection of which passages fit within the LLM's context limits while maximizing information density.

raiveFeeder — Corpus Management

raiveFeeder is a companion FastAPI application (port 8001) that decouples corpus maintenance from the main analysis engine. It shares the same ChromaDB vector store and provider ecosystem as raiveFlier — content ingested through raiveFeeder is immediately available for RAG retrieval.

• Multi-Format Document Ingestion — PDF, EPUB, TXT, DOCX, RTF, MOBI, and DJVU are converted, chunked, tagged, embedded, and stored in a unified pipeline.
• Audio Transcription — Whisper integration (local via faster-whisper or cloud via OpenAI API) transcribes interviews, DJ sets, and lectures into searchable corpus chunks.
• Image OCR Ingestion — Single-flier and multi-page scan modes extract text from historical flier archives and embed them directly into the knowledge base.
• LLM-Guided Web Crawling — An intelligent crawler scores each page's relevance (0–10) to a natural language query and predicts link relevance before queueing, narrowing crawls to topically useful content. Blind mode available for full-site harvesting.
• Content Approval Queue — Optional moderation with GitHub issue integration allows admin review before ingestion into the production corpus.
• Corpus Publishing — One API call pushes the ChromaDB tarball to a GitHub release and triggers a Render redeploy, keeping the production knowledge base in sync.