LLMs Are Not Data Cleaners (Until You Force Them to Be)

We feed our LLM receipt photos, voice memos, and text blobs. It works — but only because we don't trust it.

FoodNet accepts food input from four modalities: typed text, receipt scans, food label photos, and voice dictation. Each one is noisy in completely different ways. The first version of our pipeline sent raw input straight to an LLM and asked it to parse food items. It hallucinated "organic quinoa" from a receipt that said "ORG QN 2.99."

Here's what we built instead.

Step 1: Detect the Modality

Before any cleaning happens, we classify the input using regex signals:

# Receipt signals: price patterns, tax keywords
receipt_ocr  →  r'\$[\d.]+', r'subtotal|total|tax|visa|mastercard'

# Photo signals: nutrition label markers
photo_ocr    →  r'nutrition facts', r'serving size', r'net wt'

# Speech signals: conversational fillers
speech       →  r'\b(um|uh|like|you know|basically)\b'

# Default
text_blob    →  everything else

This detection is pure regex. No model call. It runs in <1ms and determines which cleaning pipeline activates.

Step 2: Modality-Specific Noise Removal

Each modality has its own noise signature:

Receipt OCR discards 10 pattern types:

• Bare numbers and prices ($4.99, 2.99)
• Timestamps, store numbers, loyalty card data
• Tax lines, subtotals, dividers (---, ===)
• QR/barcode artifacts

Then applies 6 fix patterns: strip remaining prices, quantity abbreviations (qty, ea, each), and special characters.

Photo OCR discards 9 pattern types:

• "Nutrition Facts", "Serving Size", "Best By" headers
• Bare measurements (150g, 200mg, 120cal)
• Ingredient list headers

Speech removes 30+ filler patterns:

• Fillers: "um", "uh", "you know", "basically", "I think"
• Commands: "can you", "I want", "don't forget", "also add"
• Quantity normalizations: "a bunch of" → "some", "like 3" → "3"

All of this runs before the LLM ever sees the input. The LLM receives clean, normalized text — not raw noise.

Step 3: LLM Parsing with Constraints

The cleaned text goes to a single LLM call (GPT-4.1-mini, temperature 0) that handles three tasks simultaneously:

1. Split multi-item text into individual items
2. Parse quantity, unit, and generic_name for each
3. Expand vague categories into concrete items

The prompt includes strict canonicalization rules:

KEEP identity-defining descriptors:
  "beef" in "beef broth", "cheddar" in "cheddar cheese"

DROP marketing/diet noise:
  organic, keto, paleo, gluten-free, non-gmo, premium, artisan

KEEP state/form only when material:
  frozen, canned, dried, smoked, pickled (NOT fresh, raw, natural)

DROP brand names:
  Heinz, Chobani, Kirkland

EXCEPT brand-as-common-name:
  Sriracha, Tabasco, Worcestershire

The receipt OCR prompt also gets abbreviation expansion rules: BEL→bell, CHKN→chicken, GRN→green, BROC→broccoli.

Step 4: Deterministic Fallback

When the LLM fails — network timeout, invalid JSON, empty results, malformed response — the system doesn't crash. It falls back to a pure-regex deterministic parser:

1. Split text on newlines, then commas
2. Run regex-based parse_food_input() on each chunk
3. Assign confidence 0.5 to all fallback results (vs. LLM's 0.7–1.0)

The confidence penalty signals downstream systems to treat these results with more skepticism. The food matcher, for example, suppresses alias creation for matches with confidence ≤0.5.

Training Data Collection

Every LLM call — input, output, modality, and latency — is logged to a training table. This collection is non-fatal (wrapped in try/except) and runs in the background. The goal: fine-tune a smaller model once we have enough examples.

We're at ~10,000 logged examples. Not enough yet, but the pipeline is in place.

The Architecture Pattern

Raw Input → Modality Detection (regex) → Noise Removal (regex) → LLM Parse → Deterministic Fallback
     ↓                                                                              ↓
  <1ms                                                                        ~500ms or 0ms

The key insight: the LLM is not the data cleaner. The LLM is the structured parser that receives already-clean data. Regex handles noise. The LLM handles ambiguity. The deterministic fallback handles failure.

None of these steps trust the previous one completely. Each has its own validation, its own confidence score, its own failure mode. The system degrades gracefully, not catastrophically.

---

Next: Why Embeddings Beat Filters (Then Don't) — our hybrid recipe search and why semantic similarity breaks on pantry staples.