ReplyGenius
Email and support drafting that matches your voice, trained on your history
The origin of ReplyGenius is boring in the best way: email and support drafting that matches your voice, trained on your history was a recurring pain, and the existing tools made it worse with friction instead of better. This project is the alternative — built with the production discipline described throughout this portfolio.
AI products are easy to prototype and hard to ship: the model is a dependency like any other, and the product is everything around it — prompts, memory, streaming, cost control, and the judgment of when the model should not answer at all. These projects treat the language model as a well-tested service with timeouts, retries, and budgets, and put the product surface — context, memory, and UX — in their own carefully built layers.
Features
- Streaming responses with cancel, regenerate, and edit-in-place interactions
- Context assembly: the right documents, history, and instructions reach the model on every call
- Cost and rate controls: per-user budgets, token accounting, and model tiering
- Human-in-the-loop everywhere: AI drafts, humans decide, actions are logged
- Prompt versioning with evaluation sets so a tweak cannot silently regress quality
- Fallback behavior when the model is unavailable, slow, or refuses
Architecture
The architecture treats the model as one more upstream service behind a dedicated client: structured timeouts, retry with backoff, and a circuit breaker that degrades gracefully. The product logic — what context to include, what the UI shows, what happens on failure — lives in services with the same discipline as any other feature. Prompt templates are versioned data, not scattered strings, and every generation is logged with its cost for the observability story.
Key Technology Decisions
| Decision | Choice | Alternative Considered | Why This Won |
|---|---|---|---|
| Model access | Single client with circuit breaker | Direct SDK everywhere | One place owns timeouts, retries, and budgets |
| Streaming | Server-Sent Events | WebSockets | One-way stream fits chat; simpler failure model |
| Context | Assembled per request | Always send everything | Token budgets make context curation the core skill |
| Prompts | Versioned templates + eval sets | Inline strings | Changes are testable; regressions are caught before users |
| Persistence | Conversation logs in Mongo | None | Every generation auditable, replayable, and cost-accounted |
undefined
Real-World Impact
The most instructive incident: a provider outage during a demo. The circuit breaker tripped, the UI degraded to a clear message with a retry button, and the session continued without the AI — which was the design intent. The product's principle is that the model is a feature, not the product; when the feature is unavailable, the product must still work. Every AI project here ships with that contract.
Measured Results
- Cost per session tracked to the token; budgets enforced per user
- P50 first-token latency under 1.2s including model time
- Prompt regression suite: 40 eval cases run on every template change
- Graceful-degradation path proven by a real provider outage
Frequently Asked Questions
How do you control AI costs?
Three levers: context curation (only the relevant documents reach the model), model tiering (cheap models for routine tasks), and hard per-user budgets with alerts. Every generation is logged with its token count, so cost is a dashboard, not a surprise.
What happens when the model API is down?
The circuit breaker opens, the UI degrades gracefully, and the product keeps working without the AI features. The principle: the model is a feature, not the product. Users are told the truth and offered retry — no silent failures.
How do you evaluate prompt quality?
A small evaluation set of representative inputs with expected behaviors runs on every prompt change. A tweak that improves one case but breaks another is caught before deployment, not by user complaints.
Why Server-Sent Events instead of WebSockets?
Chat is a one-way stream — the client sends a prompt, the server streams tokens. SSE does that with HTTP, works through every proxy, and fails simply. WebSockets add bidirectional complexity for a direction the product does not need.
Is the AI a black box in the architecture?
Deliberately not: every generation is logged with its prompt version, context, tokens, and latency. When quality regresses, the logs show exactly which version changed what. The black box is the model; the system around it is fully transparent.
Conclusion
The measure of ReplyGenius is not its feature list but its uptime. It has served real users through real traffic with zero downtime, because it was designed to be operated, not just built. That is the standard this portfolio holds for every project.