🛠️ Tech Stack

🚀 Features

• Attack Taxonomy Browser — Explore 200K+ real prompt injection attempts clustered by technique (role hijacking, instruction override, context manipulation, etc.), sourced from 4 Hugging Face datasets with difficulty ratings and dataset provenance

• Defense Sandbox — Configure a multi-layer defense stack (system prompt hardening + keyword blocklist + output filters), select attack sets by technique/difficulty/count, and run live evaluations against the Claude API with real-time SSE streaming

• Scorecard Dashboard — Animated SVG metric rings, per-technique block rate breakdowns, difficulty-stratified scoring, and worst-first bar charts that surface your weakest defenses immediately

• Guardrail Comparison Mode — Evaluate multiple defense configurations against the same attack set side-by-side, with a coverage matrix (techniques x configs) using color-coded block rates, delta arrows, and percentage-point badges

• AI Red Team Report Generator — One-click Markdown report powered by Claude that synthesizes eval data into a structured security assessment with executive summary, technique-level analysis, and actionable recommendations

• Dataset Ingestion Pipeline — Normalizes 4 heterogeneous datasets (deepset, neuralchemy, Lakera, SPML) through format-specific ingestors with German language filtering, noise detection, heuristic technique classification, and deterministic UUID-based deduplication

🧗‍♂️ Challenges

• Normalizing heterogeneous attack datasets — The 4 source datasets used different formats (Parquet vs. CSV), labeling schemes (binary labels vs. degree scores), and languages (English + German). Built a multi-layer classification pipeline with 10 primary technique regexes, CTF-specific password extraction patterns, and source-aware fallback heuristics — reducing unclassified prompts from 92% to under 6%

• Real-time SSE with late-join support — Eval runs can take minutes. Clients connecting mid-run need full event history plus live updates. Implemented a dual data structure (append-only event log + per-subscriber asyncio.Queue) with atomic writes to both, enabling seamless replay-then-stream for any connection timing

• Chakra UI v3 migration with minimal docs — Chakra v3 is a complete API rewrite from v2 (composition patterns like Card.Root/Card.Body, createSystem() replacing extendTheme()). With sparse documentation, relied on reading TypeScript type definitions in node_modules/@chakra-ui/react/dist/types/ to discover the API surface

• Composable defense pipeline architecture — Designed input/output filters as a Strategy pattern with abstract base classes, enabling the eval runner to iterate through arbitrary filter chains, short-circuit on first block, and map DefenseConfig JSON directly to filter instances — adding a new filter type requires zero changes to the pipeline

🧠 Learnings

• Data quality > data quantity — The Lakera dataset alone had 223K prompts, but aggressive noise filtering (empty/short prompt removal, casual greeting detection, level metadata mapping) was critical. Without it, casual questions like "hi how are you" polluted the attack taxonomy. Better to accept some false negatives than let noise through

• Heuristic classifiers work surprisingly well for LLM output scoring — Binary refusal detection using 12 regex patterns achieves high accuracy because LLM refusals are highly templated from training. The confidence field handles ambiguous cases without losing data points

• Deterministic UUIDs from content hashes (sha256(source:text)) enable fully idempotent re-ingestion — the pipeline can DELETE FROM then INSERT without orphaning data, and cross-dataset deduplication is automatic

• Presets are critical for portfolio demos — A recruiter's attention window is ~30 seconds. One-click presets that auto-fill the defense config and select a balanced attack set mean 2 clicks to a live result. The "Maximum" preset is tuned to show high block rates, making the first impression count