erin.dev

Back to home

🗂️ askdocs

Search your local docs with natural language — not keywords

Source Code

I built askdocs during a gap year while ramping up on LLM development, context-switching between four different library docs daily. Keyword search fails when you don't know the right terminology yet — so I built a tool that lets you point at a directory, index it once, and ask questions in plain English.

askdocs is a full RAG pipeline from scratch — no LangChain, no orchestration frameworks. It discovers files, chunks them with structure-aware splitters, batch-embeds via OpenAI, persists to a local ChromaDB vector store, and retrieves relevant passages via cosine similarity. Answers come back with exact file paths and line numbers, either as raw passages or as GPT-4 synthesized responses grounded strictly in the indexed docs.

The core technical insight: code-aware chunking using tree-sitter ASTs. Naive chunkers split on character count, which breaks functions in half — destroying embedding quality. askdocs parses Python, JavaScript, and TypeScript into ASTs and chunks on structural boundaries (function definitions, class methods, decorated definitions), producing embeddings that actually represent code semantics. This single decision measurably improved retrieval Hit@1 and Hit@3 rates across the evaluation suite.

🛠️ Tech Stack

Python 3.11ClickOpenAI API (ada-002 + GPT-4)ChromaDBtree-sitter (Python, JS, TS grammars)Pydantic + Pydantic Settingstiktokenpytest

🚀 Features

  • AST-aware code chunking — tree-sitter parses source into an AST; functions and classes become individual chunks at exact line boundaries, large classes split per-method with class signature context prefixed

  • 4-level Markdown cascade — splits on H2 → H3 → paragraphs → sentences, each level only activating if the previous produced oversized chunks, preserving section heading context in metadata

  • Dual query modes — --raw returns ranked passages with file:line citations and cosine distance scores; default mode sends passages to GPT-4 with strict grounding instructions that prevent hallucination

  • Quantitative eval suite — 20 questions across 3 doc sets with Hit@1/Hit@3 metrics, per-category breakdowns (code/text/conceptual), and timestamped JSON results for cross-run chunker comparison

  • Full CLI with management commands — index, ask, list, info, delete with progress bars, per-stage timing breakdowns, extension filtering, threshold controls, and colored terminal output

  • 117 tests across 9 test files — covering models, discovery, all chunker types, embeddings, storage, retrieval, synthesis, and end-to-end CLI integration

🧗‍♂️ Challenges

  • ChromaDB's flat metadata constraint shaped the entire data model early — no nested objects or None values allowed. I built a to_chroma_metadata() / from_chroma_metadata() serialization layer on ChunkMetadata that cleanly bridges rich Pydantic models with ChromaDB's flat key-value storage, omitting optional fields entirely rather than storing nulls

  • Chunking large classes without losing context — when a class exceeds the chunk size, splitting per-method means each method chunk loses the class identity. Solved by prefixing every method chunk with the class signature line (e.g., class DocStore:), so the embedding captures both the method's code and its class membership

  • Preventing hallucination in synthesis — GPT-4 will confidently fabricate answers if unconstrained. The system prompt enforces strict grounding: cite every claim with [path/to/file.py:L45] format, and explicitly state when the docs don't cover a topic. The --raw flag exists because I found myself verifying grounded answers against the actual passages — making the retrieval layer directly inspectable was a design decision born from using the tool myself

🧠 Learnings

  • Chunking strategy is the highest-leverage decision in a RAG pipeline. Embedding quality is bounded by what you feed the embedding model — a half-function doesn't represent its semantics. Moving from naive character-count splitting to AST-boundary splitting was the single biggest retrieval quality improvement

  • Build the evaluation suite before optimizing. Without Hit@1/Hit@3 metrics I'd be guessing whether changes helped. The eval suite with 20 curated questions across code, text, and conceptual categories made every chunker experiment measurable

  • The best developer tools give you both layers — fast access to source material and optional AI synthesis on top. The --raw flag became my most-used feature, teaching me that retrieval transparency matters as much as the final synthesized answer