I owned evaluations and observability end to end: the golden-set methodology, the CI/CD eval pipeline, and the React Native dashboard product teams used every week. The goal was to answer one leadership question with data, not opinion: were the AI changes I shipped improving the product or regressing it? I designed a golden set made of both canonical user prompts and golden set users so each run tested the model against realistic cellar profiles, not synthetic averages. I split golden questions into quantitative and qualitative tracks, piped results into a dashboard I designed and built, and made trend lines, inflection points, and regressions visible before broad rollout.

• Golden set design: I curated stable, high-value prompts and paired them with representative user accounts covering different cellar sizes, data quality patterns, and usage behaviors.
• Quantitative track: questions with objectively verifiable answers, where expected outputs were computed from hard-coded SQL queries against user profile and cellar tables. This gave deterministic pass/fail signals for metrics like bottle counts, holdings by region or variety, and other inventory facts.
• Qualitative track: recommendation and interpretation prompts where correctness is not a single database value. For these, I used an LLM-as-judge framework with explicit rubrics to score dimensions like relevance, grounding, clarity, and actionability.
• Evaluations dashboard (my build): I designed and shipped a React Native dashboard for agentic AI/ML quality operations, run history, version comparisons, quantitative pass rates, qualitative judge scores, and regression alerts in one surface leadership could read without opening notebooks.
• CI/CD and release gating: I wired eval runs into the pipeline and used dashboard trends in release reviews so prompt, retrieval, tool, and model changes shipped only when both tracks showed acceptable movement.

I designed and built this React Native dashboard as the operational home for CellarChat quality. It is not a side report; it is the surface I used with product and leadership to run agentic AI/ML evaluations, compare builds, inspect regressions, and justify release decisions. Quantitative SQL-backed checks and qualitative LLM-as-judge scores live in one place so teams could see whether a weekly change actually improved member-facing behavior.

I connected AI workflows to real user surfaces across web and mobile. CellarChat was not a side experiment; it was integrated into the core CellarTracker experience, including the collections flow where members decide what to drink and manage inventory. I worked in Python on backend AI services, in TypeScript on frontend integration, and in React Native to deliver cohesive mobile experiences, including the evaluations dashboard as a first-class internal product, not a throwaway admin page. I also contributed beyond AI-only surfaces, including subscription and payment-related improvements, because shipping successful AI products requires alignment with broader product, platform, and business workflows.

CellarChat brought AI assistance into CellarTracker's core wine experience for a platform with 1M+ members. It connected personalized recommendations and collection intelligence to daily user decisions, not just novelty chat interactions. The evaluations dashboard I built became how we proved reliability to stakeholders: visible trends, comparable runs, and a repeatable release bar instead of subjective "it feels better" reviews. Through full-stack integration, I delivered AI capabilities directly in product surfaces across web and mobile, increasing practical adoption opportunities. The work also supported broader product and revenue efforts, including subscription and payment improvements.

• Shipped production AI assistance for CellarTracker's 1M+ member platform.
• Designed and built the React Native evaluations dashboard used for weekly quality reviews, regression detection, and CI/CD-informed release gating.
• Connected AI workflows to real React Native mobile product surfaces and allowed members to thumbs up, thumbs down, and provide custom feedback about AI responses.
• Established golden-set evals with quantitative SQL checks and qualitative LLM-as-judge scoring, operationalized through the dashboard, not one-off scripts.
• Contributed to broader growth work: app rating improvements, Apple Pay subscriptions, and recurring revenue initiatives.

CellarChat made one lesson unavoidable: embeddings are the backbone of production RAG. They decide what context the model sees, what failures look like in evals, and whether weekly changes actually improve member-facing answers. The evaluations dashboard I built is what made those embedding and retrieval lessons legible to the org, every bullet below is something I could point to on a chart, not argue about in a meeting. The list is what I would tell any team shipping agentic AI on top of real user data, not theory from a notebook, but patterns that showed up repeatedly on a 1M+ member wine platform.

• Embeddings are the real interface between your product data and the model. If retrieval is wrong, no prompt engineering will save the answer.
• Wine data taught me that one embedding space must serve structured facts and unstructured language at once: bottle counts and locations live beside tasting notes, preferences, and free-form questions.
• When context windows were capped around 150k tokens, embeddings were not optional; they were how I compressed the cellar into what could fit in a single request without drowning the model in noise.
• Embedding quality is a data-modeling problem first. Region names like "Mendocino" can mean region, subregion, or appellation; bad vectors surface the wrong documents and the model confidently answers the wrong question.
• Chunk boundaries matter as much as the embedding model. Splitting tasting notes, inventory rows, and help articles poorly creates vectors that look relevant in cosine distance but fail in product context.
• Metadata on every chunk is half the retrieval system: member ID, cellar scope, vintage, storage location, and content type let you filter before you rank, which is cheaper and more reliable than pure semantic search alone.
• Hybrid retrieval beats embeddings-only in production: combine vector similarity with structured lookups and hard filters so inventory math and recommendation prompts do not compete in one undifferentiated pile of text.
• Re-embedding is a release event. Changing models, dimensions, or chunking strategy without a backfill plan silently shifts which answers your users get, even when prompts and tools stay identical.
• Stale embeddings are a silent regression. New bottles, edited tasting notes, and deleted holdings only help the assistant if your index refresh path keeps pace with member activity.
• Duplicate and near-duplicate chunks inflate recall and confuse ranking. Deduplication and canonical sources for the same fact (one cellar row, one summary chunk) improved precision more than swapping embedding providers.
• Embedding the question is not enough; embed the task. "What should I drink tonight?" and "How many bottles of Barolo do I have in Seattle?" need different retrieval profiles even when the surface wording is similar.
• Negative retrieval signals are underrated: knowing what not to pull into context (other members' cellars, outdated vintages, generic wine encyclopedia filler) reduces hallucination pressure as much as finding the right passage.
• Grounding checks should trace back to chunks, not vibes. In evals, I could only defend release decisions when failures mapped to specific retrieval misses: wrong chunk, missing metadata filter, or bad rank, not "the model felt off."
• Token cost and latency follow embedding choices. Smaller, well-curated context from strong retrieval beat sending large raw dumps; embeddings are how you buy quality per dollar at inference time.
• Fine-tuning and LoRA do not replace embeddings; they complement them. Disambiguation training helped Mendocino-style edge cases, but the assistant still needed retrieval to put the right cellar facts in front of the model on every turn.
• Quantitative eval questions exposed embedding gaps quickly: when SQL-backed golden answers failed, the bug was often retrieval (wrong slice of the cellar vectorized), not the LLM's reasoning.
• Qualitative evals still depend on embeddings. LLM-as-judge scores for relevance and grounding collapse when the retrieved set mixes generic wine advice with member-specific inventory; the judge penalizes answers that were doomed in retrieval.
• Observability for RAG means logging embedding queries, top-k results, chunk IDs, and scores, and surfacing them in an evaluations dashboard your team actually opens. Without that trail in a product you own, you cannot tell whether a bad answer was a model failure or a vector failure.
• Treat your embedding index like a product surface: version it, diff it in CI, and gate releases when retrieval metrics move, the same discipline as API contracts, because members experience retrieval as the product.
• The biggest lesson: in production RAG, embeddings are not preprocessing; they are the system. Prompts and agents orchestrate; vectors decide what truth the model is allowed to see.

CellarChat™ represents the kind of AI engineering I enjoy most: turning ambiguous product problems into reliable, evaluated, production-ready LLM systems, with a dashboard and eval discipline stakeholders can trust, not demos that only work on Tuesday.