AI Scoring Systems Drift (and How Relative Ranking Fixes It)

Our first AI scoring run rated 80% of recipes between 6 and 8 out of 10. The distribution was useless.

FoodNet uses AI-generated scores to rank recipes in search results and recommendations. The idea is simple: have an LLM evaluate each recipe on appeal and practicality, then use those scores as a ranking signal. The reality is that LLMs are agreeable raters, and absolute scoring compresses everything to the center.

Here's how we fixed it.

The Problem: Score Compression

When you ask an LLM to rate a recipe from 1-10, it tends to give everything a 6-8. The model is polite. It sees something good in every recipe. A dry, flavorless "boiled chicken breast" recipe gets a 6 because "it provides protein." A complex Thai curry gets an 8 because "it has interesting flavors."

The result: 80% of your scores are within 2 points of each other. You can't rank with that.

The Solution: Comparative Batching

Instead of scoring recipes in isolation, we batch 5 recipes per prompt and force the model to compare them:

Score these 5 recipes on appeal (1-10) and practicality (1-10).

CONSTRAINTS:
- You MUST use at least 3 different values for each dimension
- At least 1 recipe must score ≥7 on appeal (unless all are poor)
- At least 1 recipe must score ≤4 on appeal (unless all are excellent)
- Target distribution: ~15-20% of scores in 8-10, ~15-20% in 1-3

This forces the model to differentiate. It can't give everything a 7 when it must use 3+ distinct values across 5 recipes.

8 User Personas

A recipe that's perfect for a busy parent is impractical for an experienced chef. We score every recipe from 8 different perspectives:

Persona	Key Constraints
Busy Parent	<30 min, kid-friendly, one-pot preferred
Budget Cook	<$5 ingredients, batch potential
Health Runner	High protein, complex carbs, anti-inflammatory
Beginner Cook	<10 ingredients, basic techniques
Experienced Chef	Complex flavors, new techniques
Comfort Seeker	Casseroles, stews, nostalgia
High-Protein Prepper	30g+ protein, microwave-reheatable
Quick Weeknight	<20 min cook, minimal prep, pantry-friendly

Each persona has specific constraints that the model evaluates against. A 45-minute recipe automatically scores low on practicality for the Busy Parent, but might score perfectly for the Experienced Chef.

Two-Axis Scoring

Each recipe gets two scores per persona:

• Appeal (weight: 0.6) — would this persona want to make it?
• Practicality (weight: 0.4) — can they realistically make it?

composite = appeal * 0.6 + practicality * 0.4

The distinction matters. A beginner cook might find a complex French dish highly appealing (9) but completely impractical (2). Composite: 6.2 — accurately reflects the tension.

Score Scale Definitions

We provide explicit anchors to prevent drift:

Appeal:

• 1-2: Actively unappealing, violates persona constraints
• 3-4: Not interesting, generic, would scroll past
• 5-6: Decent but unremarkable
• 7-8: Appealing, matches preferences well
• 9-10: Highly appealing, strong lifestyle match

Practicality:

• 1-2: Impossible given persona constraints
• 3-4: Difficult, constraints violated
• 5-6: Doable but inconvenient
• 7-8: Practical, fits lifestyle well
• 9-10: Perfect fit, all constraints met

Normalization for Search

Raw composite scores (1-10) need normalization before they can be a search ranking signal alongside embeddings and FTS:

normalized = (avg_composite - 5.0) / 4.5

This centers the distribution at zero:

• Score 5.0 → normalized 0.0 (neutral)
• Score 9.5 → normalized +1.0 (maximum boost)
• Score 0.5 → normalized -1.0 (maximum penalty)

Unscored recipes default to 0.0 — they receive no boost and no penalty. This is critical: we can't penalize recipes that haven't been scored yet.

The Kill Switch

Two safety mechanisms:

1. Global kill switch (AI_SCORE_ENABLED): one config change disables AI scores across search and recommendations
2. Human shutoff: after a user has 5+ interactions (searches, saves, cooks), AI scores fade out. The user's own behavior becomes the ranking signal.

The reasoning: AI scores are a cold-start solution. They provide signal when we know nothing about the user. Once we know the user, their history is more valuable than any model's opinion.

Results

After switching from absolute to comparative scoring:

• Score distribution spread from 2-point range to full 1-10 range
• Clear separation between high-quality and low-quality recipes
• Persona-specific scoring surfaces the right recipes for the right context

The model still trends slightly positive (median ~5.5 rather than 5.0), but the constraints prevent the compression that made the original scores useless.

---

Next: Building a Recommendation Engine That Explains Itself — our 6-feature scoring system where every recommendation is a readable dot product.