I owned evaluations and observability end to end: the golden-set methodology, CI/CD eval pipeline, and React Native dashboard product teams used each week. They answered one leadership question with data: did a change improve the product or regress it? Canonical prompts ran against representative cellar profiles rather than synthetic averages. I split the work into quantitative and qualitative tracks, then made trend lines, inflection points, and regressions visible before broad rollout.

• Golden set: stable, high-value prompts paired with representative users across cellar sizes, data quality patterns, and usage behaviors.
• Quantitative track: hard-coded SQL against user-profile and cellar tables produced deterministic pass/fail signals for bottle counts, holdings by region or variety, and other inventory facts.
• Qualitative track: LLM-as-judge rubrics scored recommendation and interpretation answers for relevance, grounding, clarity, and actionability.
• Dashboard: I designed and shipped one React Native surface for agentic AI/ML quality operations, run history, version comparisons, pass rates, judge scores, and regression alerts.
• Release gating: eval runs fed the pipeline and dashboard trends informed reviews, so prompt, retrieval, tool, and model changes shipped only when both tracks moved acceptably.

I designed and built this React Native dashboard as CellarChat's operational home for quality. Product and leadership used it to run agentic AI/ML evaluations, compare builds, inspect regressions, and justify release decisions. SQL-backed checks and LLM-as-judge scores sit together, so a weekly change has to show its impact on member-facing behavior.

I connected AI workflows to real web and mobile surfaces. CellarChat lived in the core CellarTracker experience, including the collections flow where members decide what to drink and manage inventory. I worked in Python on backend AI services, TypeScript on frontend integration, and React Native on cohesive mobile experiences. The evaluations dashboard was a first-class internal product, not a throwaway admin page. I also contributed to subscription and payment improvements; successful AI products still have to fit the wider product, platform, and business.

CellarChat made one lesson unavoidable: embeddings are the backbone of production RAG. They shape the context a model sees, the failures an eval exposes, and whether a weekly change improves a member-facing answer. The dashboard made those lessons legible to the organization: each one below could be traced to a chart, not debated in a meeting. These are production patterns from a 1M+ member wine platform, not notebook theory.

• Embeddings are the interface between product data and the model. If retrieval is wrong, prompt engineering cannot rescue the answer.
• Wine data made one embedding space do double duty: structured facts such as bottle counts and locations alongside tasting notes, preferences, and free-form questions.
• With context windows capped around 150k tokens, embeddings compressed a cellar into one useful request instead of a costly wall of noise.
• Embedding quality starts with the data model. “Mendocino” can mean a region, subregion, or appellation; bad vectors fetch the wrong document and invite a confident wrong answer.
• Chunk boundaries matter as much as the embedding model. Poorly splitting tasting notes, inventory rows, and help articles produces vectors that look close in cosine distance but fail in product context.
• Chunk metadata—member ID, cellar scope, vintage, storage location, and content type—lets retrieval filter before ranking. That is cheaper and more reliable than semantics alone.
• Production retrieval is hybrid: vector similarity, structured lookups, and hard filters keep inventory math and recommendations from competing in one pile of text.
• Re-embedding is a release event. A model, dimension, or chunking change needs a backfill plan or it silently changes the answers members receive.
• Stale embeddings are a quiet regression. New bottles, edited notes, and deleted holdings matter only when the index refresh keeps pace with member activity.
• Duplicate and near-duplicate chunks inflate recall and muddy ranking. Canonical sources—one cellar row, one summary chunk—improved precision more than changing embedding providers.
• Embed the task, not only the question. “What should I drink tonight?” and “How many bottles of Barolo do I have in Seattle?” need different retrieval profiles despite similar language.
• Negative retrieval signals matter: excluding other members’ cellars, outdated vintages, and generic wine filler reduces hallucination pressure as much as finding the right passage.
• Grounding checks should lead back to chunks, not vibes. I could defend a release decision only when a failure mapped to a wrong chunk, missing metadata filter, or bad rank.
• Embedding choices set token cost and latency. Strong retrieval with smaller, curated context beat large raw dumps on quality per inference dollar.
• Fine-tuning and LoRA complement, not replace, embeddings. Disambiguation training helped Mendocino-style edge cases, while retrieval still put the right cellar facts in context each turn.
• SQL-backed golden answers exposed embedding gaps quickly: a failed quantitative eval was often the wrong cellar slice vectorized, not faulty LLM reasoning.
• Qualitative evals depend on embeddings too. When retrieval mixed generic wine advice with member inventory, LLM-as-judge scores for relevance and grounding fell with it.
• RAG observability means logging embedding queries, top-k results, chunk IDs, and scores in an evaluations dashboard the team actually uses. Without that trail, model failures and vector failures blur together.
• Treat the embedding index as a product surface: version it, diff it in CI, and gate releases when retrieval metrics move. Members experience retrieval as the product.
• The durable lesson: embeddings are not preprocessing in production RAG; they are the system. Prompts and agents orchestrate, but vectors decide what truth the model can see.

CellarChat™ is the kind of AI engineering I enjoy most: turning an ambiguous product problem into a reliable, evaluated LLM system that stakeholders can trust—well beyond a demo that only works on Tuesday.