F1 Hides the Thing You Actually Care About

Catalogs are full of facts that are technically present and structurally useless. A THC beverage with the SKU 10mg750well001 knows its dose is 10mg and its volume is 750ml. A description reading "20 servings per bottle" knows its serving count. None of that is queryable. It is trapped in prose and part numbers.

Getting it out is an extraction problem, and extraction is a field with forty years of evaluation practice behind it. The commerce industry has mostly decided to skip that part. Vendors ship LLM enrichment and report nothing, or report a single accuracy number with no denominator.

We built the scorer first. This is what it does and what it has told us so far, including the unflattering part.

Two layers, and why the boring one goes first

Enrichment runs in two passes with a hard line between them.

The first pass is fully deterministic. It maps what is structurally present: variants become offers, Shopify option names become attributes, money is parsed, GTINs are pulled. No model involved. Color becomes color, Flavour becomes flavor, and anything it cannot confidently map is dropped.

That last bit is enforced at the database level, which matters more than it sounds. The attribute key column has a foreign key onto a definition table with ON DELETE RESTRICT. The extractor is only permitted to emit a key that already exists in the dictionary.

That constraint is the anti-rot guarantee. Without it you get nicotine and nic_mg and nicotineStrength as three separate columns across three stores, and your cross-store normalization is dead within a quarter. The key set is global on purpose. "Flavor" has to mean the same thing everywhere or none of this compounds.

The deterministic pass also spends a surprising amount of code on denoising. Real catalogs are full of tags that look like attributes and are not: spo-default from one app, Rewardify from a loyalty integration, Smart Rebuy:* from another, campaign codes, date stamps, BFCM, friday13th. Roughly thirty regex patterns exist purely to throw things away before anything downstream sees them.

The second pass is the LLM one, and it only gets the facts the first pass structurally cannot reach. Decode the SKU. Read the prose. Infer where inference is safe.

Open keys, but not open season

The deterministic layer works against a closed key set. The LLM layer cannot, because the whole point is handling attributes nobody enumerated in advance. So it emits open-set keys with guardrails around them.

• Every attribute carries provenance. One of sku_decode, description, or inferred. These are deliberately disjoint from the deterministic layer's sources, so the two passes can never silently clobber each other on a re-run.
• Every attribute carries a real confidence, and there is a write floor. Below it, the value is dropped at write time rather than stored and hoped about.
• Keys are canonicalized. A small alias map collapses fabricType and material and fittype onto canonical forms. Deliberately small, with no governance workflow, because a big one becomes its own maintenance problem.
• Keys must match a strict pattern and there is a hard cap per product, keeping the highest-confidence ones. An open vocabulary with no cap is how you end up with 40 junk attributes on a t-shirt.
• A bad model response never throws. It returns a failed result and the product is left un-enriched. Enrichment failing closed is correct; enrichment failing loud takes a sync down.

One design note we like. Unit price is never asked of the model. It is computed from the accepted volume fact and the offer price in the persistence layer, so it inherits the provenance of the fact it derives from instead of being a second independent guess. If you can compute it, do not generate it.

The scorer

Standard information-extraction metrics, applied per gold field. Every prediction lands in exactly one of four buckets.

Precision is matches over matches plus mismatches plus hallucinations. Recall is matches over matches plus mismatches plus omissions. F1 is the usual harmonic mean.

And then hallucination rate gets reported separately, as its own number, because F1 will hide it.

This is the part most people get wrong, so it is worth being concrete. Consider a model that is careful on the fields it knows and also invents a material value for every product that does not have one. Its recall is untouched. Its precision drops a little, diluted across every field in the aggregate. F1 barely moves. Meanwhile the actual product experience is a shopper being told a rug is wool when nobody ever said it was wool.

A single averaged score is a great way to not notice the failure mode that will get you in trouble.

You cannot measure hallucination without negatives

Here is the structural problem. If your gold labels only list facts a product should have, then a prediction on some other field has nothing to compare against. You cannot tell an invention from an unlabeled truth.

So labels carry two extra pieces beyond the expected values.

A forbid list: keys this product must not have. These are hard negatives. A prediction landing on one is a hallucination by definition, and it is the only way to measure the rate honestly.

And an open-versus-closed flag. A closed label asserts its expected list is complete, so any unexpected key counts against precision. An open label, the default for silver-quality labels, treats unexpected keys as unscored. That distinction keeps an incomplete label set from manufacturing hallucinations that are really just gaps in the labeling.

Building a negative set is tedious and it is the only reason the hallucination number means anything.

What counts as a match

Commerce attributes are messy in specific ways, so the scorer runs two match schemas and reports both.

Unit handling is where naive scorers quietly lie. Values are converted to a canonical base before comparison, milligrams and kilograms to grams, centiliters and liters to milliliters. If the bases do not agree, the values are not equal, full stop. A gold value of 750 in milliliters and a prediction of 750 in grams is a mismatch, not a match, even though the numbers are identical. Getting this wrong inflates your scores on exactly the products where the model was confused.

The relaxed string rule allows token-subset matching in either direction, so "merino wool" and "wool" match. Whether that should count is genuinely arguable. It is why both schemas get printed.

Refusing to report numbers we cannot defend

Every per-field precision and recall comes with a Wilson 95% confidence interval. Wilson rather than the normal approximation, because at small sample sizes and proportions near 1.0, the normal interval produces bounds above 100%, which is a good sign you are describing noise.

There is also a support floor. Below 30 gold instances, a field is flagged as underpowered and we report raw counts instead of a rate. "3 out of 4" is honest. "75% precision" from the same data is not, and it will be screenshotted without the denominator.

Which brings us to the finding we are least excited to publish.

We run the same labeled evaluation across candidate models to pick one, on the principle that you should not assume the expensive model wins, you should measure it, and you should take the cheaper one when it lands within a few points. The ship bar is precision at or above 0.95 and recall at or above 0.85, and the sweep picks the cheapest model that clears it.

On our current labeled set, most fields sit below that support floor. The sweep is directional. It does not yet clear its own bar for making a decision, and the harness prints exactly that instead of naming a winner.

We could publish a model comparison table today. It would get shared. It would also be built on per-field samples too small to distinguish the models from each other, and we would know that when we posted it.

Why this is the whole game

Enrichment quality is not an internal engineering concern that shoppers never see. It is upstream of everything.

As we wrote about where agent filters actually read from, the structured surfaces that AI assistants query run on constrained vocabularies. Getting a value into one of those is a classification problem against a closed value set, not free-form generation, which is the good news: it structurally removes the invented-value failure mode. But it also means a wrong value is not a soft error. It is a product appearing under a filter it does not belong to, or vanishing from one it does.

And when the same data feeds a sales conversation, a wrong attribute is worse than a missing one. A shopper told nothing asks a follow-up. A shopper told the wrong fabric buys the thing and returns it.

That asymmetry is why the ship bar is lopsided. Precision at 0.95 and recall at 0.85. We would rather leave a field empty than fill it with something plausible.

None of this is novel as methodology. It is standard extraction evaluation, the kind the research literature has run for years. The only unusual thing is applying it to commerce catalogs and publishing the result, including when the result is that our sample is too small to say.

If you want to see how your own catalog scores on the parts that are visible from outside, Kinect's UCP test reads your public product records and grades description depth, checkable facts, variant legibility, and structured attributes. If you would rather talk through what enrichment looks like on your catalog, book a demo.