BroncoBot
Technical Architecture
How BroncoBot retrieves and grounds answers from 72,499 chunks of official CPP content.
Architecture Overview
User Question
Natural language query
Claude Sonnet
Tool-calling LLM
search_corpus
Single retrieval tool
BM25 Keyword
In-memory, 72K chunks
Gemini Semantic
pgvector, 768d vectors
Hybrid Merge
70% semantic + 30% BM25
Top 8 Results
With source URLs
Grounded Answer
Cited response
Key Technical Decisions
| Decision | What we chose | Why |
|---|---|---|
| Search method | Hybrid BM25 + semantic | BM25 catches exact matches (course codes, names); semantic catches paraphrased intent |
| Embedding model | Gemini text-embedding-004 (768d) | Free tier, RETRIEVAL_QUERY task type optimizes for search |
| Vector storage | Supabase pgvector (HNSW index) | 72K vectors exceed Vercel's 250MB bundle limit; HNSW gives ~10-50ms query time |
| BM25 location | In-memory (serverless) | 70MB of chunk data fits in Vercel's limit; no DB round-trip for keyword search |
| Hybrid weights | 70% semantic / 30% BM25 | Validated via blind evaluation (0.95 MRR) |
| Result count | 8 per query | A/B tested against 15 — higher MRR at 8 due to candidate pool distortion |
| Tool architecture | Single tool (search_corpus) | Tested 5-tool approach; it routed queries to sparse structured data and degraded answers |
| LLM integration | Tool-calling, not RAG injection | LLM decides when to search; avoids context bloat and hallucination from injected chunks |
Corpus Statistics
8,042
Total pages indexed
72,499
Total chunks
72,499 (100%)
Chunks embedded
768
Embedding dimensions
~110K
Unique BM25 terms
~600 chars
Avg chunk length
Retrieval Evaluation — Blind A/B Test (10 queries)
We validated our search pipeline with objective metrics, not subjective judgment.
| Metric | limit=8 | limit=15 |
|---|---|---|
| Mean Top-1 Score | 0.7388 | 0.7472 |
| Mean Reciprocal Rank (MRR) | 0.9500 | 0.8833 |
| Queries with correct answer in top-3 | 9/10 | 8/10 |
Key finding: Expanding the semantic candidate pool from 24 to 45 (via match_count: limit * 3) pulled in tangentially related chunks that distorted hybrid score normalization. 6/10 queries returned different top-3 rankings. We chose limit=8 for higher MRR despite marginally lower top-1 scores — correct ranking matters more than marginal relevance gains.