What I Got Wrong Building an AI System (So Far)
I shipped 42 database migrations, 2 AI agents, and a recommendation engine. Here's what I'd do differently.
Building FoodNet has been a crash course in where AI systems fail silently. Not the dramatic failures — model timeouts, API errors, budget overruns — but the quiet ones. The ones where the system keeps running, keeps returning results, and slowly poisons its own data.
Mistake 1: Fuzzy Matching That Poisoned Its Own Cache
The food matcher uses RapidFuzz for fuzzy string matching as a fallback. When a fuzzy match succeeded, we cached it as an alias for faster future lookups. This seemed like a clear optimization.
The problem: "apple" fuzzy-matched to a cheese variant at 87% confidence and got cached. Every subsequent "apple" query returned cheese — instantly, confidently, from the alias cache. The system had learned the wrong thing and was now serving wrong answers faster.
The fix was one threshold: only persist fuzzy matches with scores ≥99 as aliases. Below that, return the match but don't cache it. The system stopped auto-learning from low-confidence matches.
Lesson: Caching AI outputs without confidence thresholds is a data poisoning vector. Your system shouldn't learn from guesses.
Mistake 2: Sending Raw OCR to an LLM
The first version of our input pipeline sent raw receipt OCR text straight to the LLM parser. The LLM was asked to extract food items from noisy text.
A receipt with ORG QN 2.99 (organic quinoa, $2.99) came back as "organic quinoa". Sounds correct. But the receipt actually said ORG QN — no spaces, no context. The LLM inferred "organic quinoa" and presented it with full confidence. Another receipt's BEL PPR 1LB (bell pepper, 1 pound) came back as "Belgian pepper".
The model wasn't wrong in a way you could catch with validation. It returned plausible food items. The hallucinations were semantically valid but factually incorrect.
The fix: regex-first cleaning. Strip prices, quantities, and metadata before the LLM ever sees the text. Add abbreviation expansion rules to the prompt (BEL→bell, PPR→pepper, ORG→organic). The LLM now receives cleaner input and produces fewer hallucinations.
Lesson: LLMs fill gaps with plausible fiction. Remove the gaps before asking the model to parse.
Mistake 3: Embedding Everything
Our recipe search uses OpenAI embeddings for semantic similarity. The initial implementation embedded the full recipe: title, description, all ingredients. The search worked — sort of.
Then we noticed that "chicken soup" returned pasta recipes. And "Thai curry" returned Italian dishes. And almost everything returned something with garlic.
The reason: salt, oil, butter, flour, eggs, garlic, and onion appear in 70%+ of recipes. When you embed the full ingredient list, these ubiquitous items dominate the vector. Every recipe looks similar in embedding space because they all share the same base.
The fix: two exclusion lists. HARD_DROP (25 items like salt, water, cooking spray) are always excluded. BORDERLINE (95 items like oils, butter, flour, garlic) are demoted to the end of the ingredient list unless the recipe title reinforces them ("Garlic Butter Shrimp" keeps garlic and butter).
Lesson: Embeddings capture what you give them. If most of your signal is noise, the embedding is noise.
Mistake 4: Author Names in Embedding Space
A subtler embedding problem: "Mary's Famous Chicken Soup" clustered with "Mary's Italian Pasta" instead of with other chicken soups. The embedding was giving weight to the author's name as a semantic signal.
This affected any recipe with a possessive in the title. The embedding space had implicit author clusters that had nothing to do with food similarity.
The fix: strip possessives from titles before embedding. "Mary's Famous Chicken Soup" → "Chicken Soup".
Lesson: Test your embedding similarities manually. Look at what's actually close in the vector space, not just what the search results look like.
Mistake 5: AI Scoring Without Constraints
Our first recipe scoring pipeline asked GPT-4 to rate each recipe 1-10 on appeal and practicality. The result: 80% of scores landed between 6 and 8. The model was being polite. It could find something nice to say about every recipe.
This made the scores useless for ranking. A 2-point spread across 2,000 recipes gives you no meaningful differentiation.
The fix: comparative batching. Send 5 recipes per prompt. Force the model to use ≥3 different values per dimension. Require at least one score ≥7 and one ≤4 per batch. Add 8 user personas with specific constraints so "appeal" means something different for a Busy Parent vs. an Experienced Chef.
Lesson: LLMs are agreeable by default. Absolute scoring produces gaussian blobs. Relative scoring with explicit constraints produces usable distributions.
Mistake 6: Building Agents Before Having Data
I built two full AI agents early on: a Food Improver (for enriching food metadata) and an Accuracy Improver (for fixing bad matches). Each has its own services, repositories, schemas, job tracking, flagging system, and admin UI.
The agents work. They're well-engineered. But they were built before I had enough data to justify the complexity. A simple script that ran the same logic in a loop would have been 90% as effective at 10% of the code.
Lesson: Don't build infrastructure for scale you don't have yet. A script is fine. A cron job is fine. Build the agent when the script can't keep up.
Mistake 7: Not Tracking Events From Day 1
The recommendation engine was initially built without event tracking. We could recommend recipes but couldn't measure if anyone clicked, saved, or cooked them.
When we finally added the engagement ladder (shown → clicked → saved → cooked), we had to backfill event wiring across the entire frontend — Dashboard, RecipePanel, RecipeDetail, HeroCards. Every component that surfaced a recommendation needed to emit events.
This was 42 migrations worth of iteration that could have been 3 if we'd built the event table first.
Lesson: If you're going to build a recommendation system, instrument it from day one. The data you collect in the first month shapes every decision after that.
The Meta-Lesson
Every mistake here shares a pattern: the system worked, but it was quietly wrong. Fuzzy matches returned food items. OCR parsing returned structured data. Embeddings returned search results. Scores returned numbers.
The failures were invisible at the surface level. They only showed up in the data: wrong aliases accumulating, embedding clusters that didn't make sense, score distributions with no variance.
The fix is always the same: add constraints, add thresholds, add measurement. Don't trust AI outputs without validation. Don't cache without confidence gates. Don't score without distribution requirements. Don't recommend without event tracking.
Production AI systems aren't about making the model smarter. They're about making the system honest about what the model doesn't know.
This is post 6 of 6 in the FoodNet engineering series. Start from the beginning: Where AI Actually Belongs in Production Systems.