CivicShield AI
Shipped APPLICATIONA civic assistant that would rather refuse than hallucinate: retrieval-before-generation, a fail-closed evidence policy, reviewer queues, and a Known-Limitations register that became my documentation standard.
> attest CivicShield AI
| claim | status | evidence |
|---|---|---|
| Refuses to answer when verified evidence is insufficient — pinned in the model's response contract | SHIPPED | src/lib/ai/prompts.ts ↗ |
| Policy and retrieval test suites run through a zero-dependency custom runner | TESTED | tests/run.ts ↗ |
| README carries five Mermaid architecture diagrams plus a Known Limitations register | DOCUMENTED | github.com/ahmadrrrtx/CivicShield-ai ↗ |
01Why civic answers deserve different engineering
When an assistant explains a benefit program or an urgent support pathway, a confident wrong answer isn't a bug report — it's someone skipping a deadline. So the product constraint came first: the system's default output is "I can't verify this," and every answer must earn its way past that. A 48-hour build, deliberately disclosed: fast because the policy was clear beforehand, not because anything was skipped silently.
02The flow: generation is the last step, not the first
Questions enter retrieval over official-source documents; candidates are validated against Zod
schemas into VerifiedEvidenceItems; the model may only see that list, must answer in a typed
JSON contract with per-citation traceIds, and its output is refused if it cites anything outside the
evidence set. Low-confidence fallback isn't UI sugar — it's the policy's observable surface.
'You must never fabricate facts, citations, departments, dates, URLs, eligibility rules, or trace identifiers.',
'You must never claim certainty when evidence is incomplete.',
'Only use the verified evidence provided to you. Do not cite any source outside that evidence list.',
'If verified evidence is insufficient, set grounded to false, confidence to 0.2 or lower, include the sentence "I cannot confidently verify this information." inside text, and keep citations empty.',03What's real under the hood
- Typed provider registry (
src/lib/ai/providers/) with fallback — Groq wired as primary, and the abstraction is the reason a second provider is a config change. - Conversation state in a single honest 22.8 KB module rather than a framework of interfaces about an interface.
- Policy + retrieval test suites behind a zero-dependency runner (
tests/run.ts) — pass/fail in CI terms, not vibes. - Rate limiting on chat, encrypted provider-key storage, security headers; reviewer queue + audit trail for refused queries.
- Five Mermaid architecture diagrams in the README — the same files this site's diagram lane restyles.
Known limitations — the section that started as an apology and became the standard
- Better Auth is wired but email verification / OAuth are not enabled. MVP honesty, documented.
- One active AI provider; the registry's fallback path is tested in isolation, not in traffic.
- No live deployment — the case study says so instead of shipping a 502 button. Retrieval quality is benchmarked on a seeded corpus, not the open web.
- Built in 48 hours with heavy AI-tool assist; what I own is the policy design and every reviewed line.
04Aftermath
The Known-Limitations section here became a rule for every repo I touch, including this site's own verifier: if the docs admit the weak parts first, readers trust the strong parts too. That's also why the ledger above this essay lists six claims and not forty adjectives.