AI Chatbot
Build a production AI chatbot with RAG, function calling, and edge deployment.
Design & Architecture
Choose model, design RAG retrieval, function-calling schema, and conversation state.
Why
Chatbot architecture decisions are sticky: your retrieval index, tool schema, and state shape all couple to the model's context window and API. A wrong retrieval strategy (semantic-only vs keyword+semantic) caps answer accuracy at ~60% no matter how good the prompt. Design the data flow before the prompt.
When
Before writing the chat endpoint. 1-2 weeks to validate retrieval quality on real questions before building the UI.
How
1. Choose the model + provider
- OpenAI gpt-4o / Anthropic Claude / open Llama 3 via Together
- Stream tokens (SSE) for perceived latency; budget context window
- Pick by task: reasoning (Claude), tool-use (gpt-4o), cost (Llama)
2. Design RAG retrieval
- Chunk documents (500-1000 tokens, 50-token overlap)
- Embed + store in a vector DB (pgvector, Pinecone, Qdrant)
- Retrieve top-k (5-10), rerank with a cross-encoder
- Hybrid: BM25 keyword + semantic — beats semantic-only on exact terms
3. Design function calling (tools)
- Define a JSON schema per tool (name, description, params)
- Tools the model can call: search_docs, get_order, create_ticket
- Keep tool descriptions precise — the model reads them to decide
4. Design conversation state
- Store messages in a DB (not just session memory) for audit
- Summarize old turns when approaching the context limit
- Track tool calls + results so you can debug failures
5. Plan evaluation
- Build a golden set of 50-100 Q&A pairs from real users
- Measure retrieval recall + answer faithfulness (RAGAS / TruLens)
- Eval before shipping — "vibes-based" chatbot testing misses regressions
Example
Architecture:
User msg -> retrieve(k=8) -> rerank -> build prompt -> stream LLM
-> tool_call? -> exec -> feed back
Vector store (pgvector):
docs (id, content, embedding vector(1536), metadata jsonb)
index: ivfflat on embedding
Tool schema (OpenAI format):
{ "type": "function", "function": {
"name": "get_order", "description": "Look up a user's order by ID",
"parameters": { "type": "object", "properties": {
"order_id": { "type": "string" } }, "required": ["order_id"] } } }
Golden eval set: 80 Q&A in eval/golden.jsonl, run RAGAS on every PR
Common Problems
- ⚠Semantic-only retrieval — misses exact matches (error codes, product IDs); add BM25
- ⚠No reranking — top-k by cosine includes near-duplicates; a cross-encoder rerank lifts accuracy 10-20%
- ⚠Vibes-based eval — looks fine in demos, regresses silently when the index or prompt changes
Best Practices
- ✓Use hybrid retrieval (BM25 + semantic) and rerank with a cross-encoder — it's the single biggest accuracy lever
- ✓Define tool schemas as precisely as a typed function — vague descriptions cause wrong tool calls
- ✓Build a golden eval set before the UI; run RAGAS in CI so retrieval regressions fail the build
Resources for this stage
Tools
Templates
Implementation & Streaming
Build the chat endpoint with SSE streaming, tool execution loop, and guardrails.
Why
Streaming tokens via SSE cuts perceived latency from 5s to <500ms — the difference between 'feels smart' and 'feels broken'. The tool-call loop (model calls tool -> you execute -> feed result back) is where most chatbot bugs live: infinite loops, unvalidated tool args, and context overflow.
When
After retrieval quality is validated on the golden set. Build the streaming endpoint + tool loop, then the UI.
How
1. Stream responses with SSE
- res.setHeader('Content-Type', 'text/event-stream')
- Stream tokens as they arrive; send [DONE] at the end
- Frontend: EventSource or fetch + ReadableStream
2. Implement the tool-call loop
- Loop: call model -> if tool_calls, execute each -> append results -> call model again
- Cap iterations (max 5) to prevent infinite loops
- Validate every tool arg with zod before executing
3. Add guardrails
- Input: filter prompt injection / PII before retrieval
- Output: check the answer is grounded in retrieved context (no hallucination)
- Rate limit per user; cap tokens per turn
4. Manage context window
- Token-count every message; summarize when > 70% of limit
- Always keep the system prompt + latest 2 turns verbatim
- Drop retrieved chunks that didn't make it into the answer (for cost)
5. Persist + observe
- Store every turn (message, tool calls, retrieved docs) in the DB
- Log to LangSmith / Langfuse for tracing and eval
- Track: tokens in/out, latency per turn, tool-call success rate
Example
SSE streaming endpoint (Node):
res.writeHead(200, { 'Content-Type': 'text/event-stream' });
const stream = await openai.chat.completions.create({ stream: true, ... });
for await (const chunk of stream) {
res.write(`data: ${JSON.stringify(chunk.choices[0].delta)}\n\n`);
}
res.write('data: [DONE]\n\n'); res.end();
Tool-call loop (capped):
for (let i = 0; i < 5; i++) {
const res = await callModel(messages);
if (!res.tool_calls?.length) return res.content;
for (const call of res.tool_calls) {
const args = schema.parse(JSON.parse(call.arguments)); // validate
messages.push({ role: 'tool', tool_call_id: call.id, content: await runTool(call.name, args) });
}
}
Common Problems
- ⚠No iteration cap on the tool loop — a model that re-calls the same tool loops until timeout
- ⚠Buffering the full response before sending — 5s blank screen, users assume it's broken
- ⚠No grounding check — the model answers confidently from training data, not your retrieved docs
Best Practices
- ✓Stream tokens via SSE from the first token — perceived latency is the #1 chatbot UX metric
- ✓Cap the tool-call loop (max 5 iterations) and validate every tool arg with a schema before executing
- ✓Check the answer is grounded in retrieved context; flag ungrounded claims rather than showing them
Deployment & Monitoring
Deploy to Vercel/Cloudflare with streaming, cost tracking, and quality alerts.
Why
Chatbot costs are unbounded — one user in a loop can burn $100 of tokens in an hour. Without per-user rate limits, cost dashboards, and quality alerts (hallucination rate, tool failure rate), a chatbot degrades silently and expensively. Edge deployment (Cloudflare Workers / Vercel) keeps streaming latency low globally.
When
When the endpoint passes the golden eval set. Deploy behind a feature flag, ramp to 10% -> 50% -> 100% of users.
How
1. Choose the platform
- Vercel (Next.js Route Handlers): easy, streams SSE, edge runtime option
- Cloudflare Workers: global edge, generous free tier, AI bindings
- Fly.io: if you need a long-running Python/Go backend with a vector DB
2. Deploy with streaming
- Edge runtime for lowest latency (but watch context-window limits)
- Keep the SSE connection alive; handle client disconnects (abort the model call)
3. Track cost per conversation
- Log tokens in/out per turn; multiply by model pricing
- Alert if avg cost/conversation spikes > 2x baseline
- Set per-user daily token budget; hard-stop or downgrade model on exceed
4. Monitor quality
- Sample 5% of conversations for human review (or LLM-as-judge)
- Track hallucination rate, tool-call failure rate, user thumbs-down rate
- Alert if thumbs-down > 10% in a 1h window
5. Iterate safely
- A/B test prompt/index changes on 10% of traffic
- Re-run the golden eval set on every deploy (block on regression)
- Version your prompts (git) so a bad change is a one-line revert
Example
Vercel Route Handler (edge + streaming):
export const runtime = 'edge';
export async function POST(req) {
const stream = await chat(req.body); // returns ReadableStream
return new Response(stream, {
headers: { 'Content-Type': 'text/event-stream' } });
}
Cost tracking (per turn):
const cost = (promptTokens * 0.005 + completionTokens * 0.015) / 1000;
await db.turns.insert({ userId, cost, tokens: { in, out } });
// Alert: SUM(cost) per user per day > $2 -> downgrade to gpt-4o-mini
Quality alert (Langfuse):
thumbsDownRate(window=1h) > 0.10 -> PagerDuty -> investigate prompt diff
Common Problems
- ⚠No per-user cost cap — one user in a tool loop burns the month's API budget in an afternoon
- ⚠Deploying without re-running the golden eval — a prompt tweak regresses retrieval and ships to all
- ⚠Blocking on the full response instead of streaming — high latency, users leave before the answer
Best Practices
- ✓Set a per-user daily token budget and downgrade (not block) on exceed — protect margin without ruining UX
- ✓Run the golden eval set as a CI gate on every prompt/index change; block deploys on regression
- ✓Deploy to edge (Vercel/Cloudflare) and stream from token one — latency is the #1 chatbot metric
Resources for this stage
Tools
Templates
Full Developer Lifecycle
A 14-stage developer roadmap from idea to maintenance — visualize dependencies and track progress.
Next Step