Catalog Scout 1.0.3: measured detection accuracy
Catalog Scout's detail checks read product wording: size, material, care/use, and the furniture and textile variants. This file reports how often they are right on labelled test products. To reproduce the numbers, run npm run evaluate in source/. To score another version, run node evaluate.mjs --engine <other core.mjs>. Before and after any rule edit, run npm run gate (see AI-OPERATOR.md).
How it is measured
- Labels: each test product has truth labels written from its copy alone. Is a measured size stated? Is a material named? Is a specific care or use instruction given? The labels are in
source/fixtures/. - Decisions: the real
audit()runs end to end on CSV input, and every check on every product is one decision. An advisory ("No material stated") is correct when the fact really is missing, and a false alarm when it is stated. - Advisories correct: the share of advisories that point at a real gap. This is what your client sees.
- Real gaps caught: the share of truly missing facts that got an advisory.
The number to quote: blind set, first and only run
The builder wrote a new blind set of 58 products (22 décor, 18 furniture and 18 textiles) and froze the labels in a commit before changing any 1.0.3 rule. The rules were then frozen in their own commit before the set was run. It was scored once with both engines, and nothing was tuned afterwards. The full record is in source/fixtures/blind-1.0.3/RESULTS.md.
| Family | Engine | Advisories correct | Real gaps caught | Accuracy |
|---|---|---|---|---|
| All (58) | 1.0.3 | 96.1% (49/51) | 98% (49/50) | 98.4% |
| Décor (22) | 1.0.3 | 100% (18/18) | 100% (18/18) | 100% |
| Furniture (18) | 1.0.3 | 94.4% (17/18) | 100% (17/17) | 98.6% |
| Textiles (18) | 1.0.3 | 93.3% (14/15) | 93.3% (14/15) | 96.3% |
| All (58) | 1.0.2 | 75.9% (41/54) | 82% (41/50) | 88.5% |
| Décor | 1.0.2 | 76.5% (13/17) | 72.2% (13/18) | 86.4% |
| Furniture | 1.0.2 | 66.7% (16/24) | 94.1% (16/17) | 87.5% |
| Textiles | 1.0.2 | 92.3% (12/13) | 80% (12/15) | 92.6% |
Why the new numbers are probably optimistic.
- The same session wrote everything. This set was written by the builder session after it had read the reviewers' failing phrasings, and that session also wrote the 1.0.3 rules. It is blind to tuning, but not independent.
- The set is small: 58 products and 50 true gaps. One error moves a family's figures by 5–7 points.
- Expect lower on real catalogs. On catalogs nobody has seen, expect lower figures until an independent test says otherwise.
The old engine scored about the same here as in the reviewer's own test. The functional reviewer's own blind test of 1.0.2 found 72.1% of advisories correct and 79.5% of gaps caught, with furniture at 64%. On this set, 1.0.2 scores 75.9% / 82% overall and 66.7% for furniture. That similarity suggests the set is not unusually easy.
The three blind-set errors are left in place:
- Ash console: "refresh the finish with furniture wax" is not read as care. This is a false alarm.
- Play mat: "wipes clean with a damp cloth" is not read as textile wash care. This is a false alarm.
- Percale sheets: "percale" is a weave, but it counts as a fiber, so the missing fiber content goes unflagged. This is a missed gap.
Sets used while repairing (not accuracy estimates)
These sets were used to find and fix failures, so a high score on them shows the reported problems are fixed. It is not an estimate of accuracy on unseen catalogs.
| Set | Products | 1.0.2 | 1.0.3 |
|---|---|---|---|
| 1.0.2 reviewers' CSVs and probes (functional décor/furniture/textiles, 14 probes, commercial doormat CSV) | 64 | 76.3% correct (45/59) · 83.3% caught (45/54) | 100% (54/54) · 100% (54/54) |
| Earlier reviewers' fixtures | 26 | 100% · 100% | 100% · 100% |
| Builder set, 3 families | 60 | 100% · 100% | 100% · 100% |
| Former held-out set (blind for 1.0.2; used for fixes since) | 35 | 100% correct · 97.6% caught | 100% · 100% |
Label changes in 1.0.3, disclosed:
- Round pieces: three products (builder f09 and held-out h21 and h25) were relabelled so that "diameter + height" counts as full overall dimensions for a round piece, as the functional reviewer asked.
- Transcribed labels: the commercial reviewer's doormat CSV was labelled by the builder from the reviewer's written findings. Ambiguous checks are left unscored.
What the checks do and don't do
- Stated, not true: they tell you whether a fact is stated, never whether it is true. Check a sample of findings on every new client catalog.
- Line-sheet drafts are not guesses. Each one is copied from a cell that passes the same detector, and it cites the row and column. It is still only as current as the vendor's file.
- When a phrasing is misread: add that product to
source/fixtures/with its correct label. Then runnpm run gate. A rule edit that creates any new false alarm or missed gap fails the gate.