# Losing It food index: contract v2 This is the single source of truth shared by the data pipeline (Python, `FoodIndex/scripts/`) and the iOS app (Swift, `LosingIt/Engine/`). If code and this file disagree, this file wins; change both sides together. **What changed from v1 (2026-11-15):** - Stemming: `ies` → `y` now needs 5+ letters, and singular forms ending in `ie`, `che`, `oe` and `ze` are folded too. "cookie" and "cookies", "pie" and "pies", "brownie" and "brownies", "quiche" and "quiches" now match each other. - Search: soft query words (numbers and amounts like "100g", single characters, portion words like "cup" or "half") are not required. Candidates come from every required word's list, not only the shortest. The ranking adds neutral words, USDA class prefixes, a first-segment bonus, a soft-word bonus and a penalty for names of derived products ("Fry sauce" for "fries", "meatless" for "chicken"), and ties break by fewer name words, then the shorter name. - Offline hits must contain every word the online search required, so a brand only the index knows can't be silently ignored by the offline half of the merge. - The app's as-you-type behaviour is written down (informative): the unfinished last word, when the online search falls back to the finished words, and how online results and offline hits are narrowed to it. - The source letter `L` is reserved. Sources may carry a short `credit` for in-app display. Duplicate hiding is written down. The hosting layout, record format, hash, buckets and barcode rules are unchanged from v1; the manifest stays `"format": 1` and adds `"contract": 2`. ## Why it looks like this The index is a set of static JSON files on Cloudflare Pages (free plan: up to 20,000 files, 25 MiB each; static requests are free and unlimited). There is no server code and no database, so there are no daily query limits. The phone fetches only the few files a search needs and does the matching and ranking itself. The server only ever sees a bucket number, never the typed text. ## Hosting - Base URL: `https://losingit-food.pages.dev`. The app keeps it in one constant. - `/manifest.json`: small, cached for 1 hour. It points at the immutable data folder for the current build: ```json {"format":1,"contract":2,"build":"20261115","path":"d/20261115","tokenBuckets":12000,"barcodeBuckets":4096, "maxPerToken":1000,"counts":{"records":0,"bySource":{"S":0}}, "sources":[{"code":"S","name":"USDA FoodData Central (SR Legacy)","license":"CC0 1.0","url":"https://fdc.nal.usda.gov/", "attribution":"…","credit":"…","generic":true}]} ``` - `//t/.json`: token shard. `` is a decimal integer with no padding. - `//b/.json`: barcode shard. - A missing shard file (HTTP 404) means an empty bucket. The app must treat 404 as "no entries", not as an error. - `/404.html` must exist. Without it Cloudflare Pages answers every missing path with `index.html` and status 200. - `/attribution.html`: human-readable credits and licenses for every source, including the ODbL notice. `/index.html`: a short landing page. `/scripts/`: the published build scripts (ODbL 4.6). - `/_headers`: `/d/*` gets `Cache-Control: public, max-age=31536000, immutable`; `/manifest.json` gets `Cache-Control: public, max-age=3600`. - Files are plain JSON (UTF-8, no BOM, compact separators). Cloudflare compresses them on the fly; URLSession decodes that. - The whole site stays at 19,500 files or fewer, and every file is under 5 MiB (aim for under 400 KB raw). ## Record (JSON array, positional) `[id, name, brand, kcal, unit, portions, barcode, src, score]` | # | field | rules | |---|---|---| | 0 | id | `":"`, e.g. `"S:171705"`, `"B:2345678"`, `"O:3017620422003"`. ASCII only. Unique across the whole index. | | 1 | name | Display name. Trimmed, whitespace collapsed, 2–120 characters. USDA ALL-CAPS branded names are converted to readable title case (keep units like "oz", "g" and "%" sensible). | | 2 | brand | The brand, or `""` for generic foods. | | 3 | kcal | kcal per 100 g (unit `g`) or per 100 ml (unit `ml`). Number rounded to 1 decimal; 0 ≤ kcal ≤ 900. | | 4 | unit | `"g"` or `"ml"`. | | 5 | portions | `[[label, amount], …]`, most useful first, at most 6. `amount` is grams (or ml when unit is `ml`) for one of that portion and must be > 0 and ≤ 5000. `label` is human text **without** the gram amount, e.g. `"1 medium"`, `"1 cup, sliced"`, `"1 bar"`, `"1 serving"`. Do **not** include a plain 100 g/100 ml portion; the app adds it. Can be `[]`. | | 6 | barcode | Normalized barcode (see below) or `""`. | | 7 | src | Single uppercase letter: `F` USDA Foundation, `S` USDA SR Legacy, `N` USDA FNDDS (survey foods), `B` USDA Branded, `O` Open Food Facts. Extra sources get their own unused letters, listed in the manifest. `L` is **reserved** for the user's own log in the app and never appears in the index. | | 8 | score | Integer 0–1000, the default rank (quality plus popularity). Higher is better. Brand-only names and other build-time judgements are folded in here, not in the search ranking. | Generic sources are `F`, `S`, `N` and any extra generic (non-branded) source the manifest marks `"generic": true` (never `B`, `O` or `L`). **Duplicate hiding:** a record with a barcode may be left out of every token list when another record with the same normalized name, brand and unit and kcal within 2% is kept (the higher (score, then lower id) one is kept). Hidden records stay in the barcode shards, so scanning still finds them. ## Token normalization (identical in Python and Swift) Given a string: 1. Apply Unicode NFKD, then remove every scalar whose General_Category is `Mn` (nonspacing mark). 2. Lowercase it (Python `str.lower()`, Swift: the full lowercase mapping plus Python's Final_Sigma rule). 3. Split into **pieces** on every scalar whose General_Category is **not** a Letter (`L*`) or Number (`N*`). 4. Drop pieces shorter than 2 scalars. 5. Light English stemming, only when the piece is all ASCII `a`–`z` and at least 4 characters long (`n` = its length). Apply the **first** rule that matches, once: 1. ends with `ies` and `n ≥ 5` → replace `ies` with `y` (berries → berry, fries → fry, cookies → cooky) 2. ends with `sses` → drop the final `es` (glasses → glass) 3. ends with `oes`, `ches`, `shes`, `xes` or `zes` → drop the final `es` (tomatoes → tomato, peaches → peach, quiches → quich) 4. ends with `ie` → replace `ie` with `y` (cookie → cooky, brownie → browny, brie → bry) 5. ends with `che`, `oe` or `ze` → drop the final `e` (quiche → quich, glaze → glaz, shoe → sho) 6. ends with `s`, but not with `ss`, `us` or `is` → drop the final `s` (eggs → egg, oats → oat, pies → pie) 6. Remove stopwords (exact match, after stemming): `and with of the in an for or to on de la le et en con mit und from`. 7. Remove duplicate tokens, keeping the first occurrence. `normalize(s)` is the resulting token list. The indexed text for a record is `name + " " + brand`. ## Hash and buckets FNV-1a, 32-bit, over the UTF-8 bytes of the token (or of the normalized barcode): `h = 0x811C9DC5; for each byte b: h = ((h XOR b) * 0x01000193) mod 2^32`. - token bucket = `h % tokenBuckets` - barcode bucket = `h % barcodeBuckets` ## Barcode normalization Keep ASCII digits `0`–`9` only. Length 12 (UPC-A) → prefix `"0"`. Length 14 starting with `"0"` → drop the first digit. Lengths 8 and 13 stay as they are. Any other length from 6 to 14 stays as it is; anything else is invalid. The app expands UPC-E to UPC-A first (`FoodLookup.expandUPCE`) and only looks up 8–14 digit codes. ## Token shard: `{"w": {"": [i, …]}, "r": [record, …]}` - Every token whose bucket is this shard has a list of indices into `r`, sorted by score descending (ties by id ascending), capped at `maxPerToken`. - `r` holds each referenced record once per file. ## Barcode shard: `{"": record, …}` One record per barcode. When sources collide, prefer `B` (manufacturer label data) over `O`; within a source, prefer the higher score. A UPC-E code may also be filed under its UPC-A form (as a 13-digit key), pointing at the same record. ## Offline bundle in the app: `LosingIt/Resources/GenericFoods.json` `{"format":1,"contract":2,"build":"…","sources":[…],"r":[record, …]}`. Only generic records (barcode `""`). The app builds its token index in memory with the same normalization. Target size: under 4 MB. ## Reference search (the app implements this in Swift; QA has a Python twin) ### Word lists - `NEUTRAL` = `raw nfs ns plain regular fresh whole cooked prepared` - `SOFT` = `slice piece cup serving small medium large oz ounce gram ml glass can bottle scoop tbsp tsp tablespoon teaspoon portion lb kcal calory half quarter dozen handful pint quart liter litre packet fl pinch` (these are tokens, i.e. already stemmed: "slices", "cups", "calories" all hit them) - `UNITS` = `g gr kg mg oz ml l cl dl lb lbs cal kcal ct pc pcs in inch` - `CLASS` = `fish beef pork lamb veal chicken turkey cheese yogurt pepper nut seed beverage crustacean mollusk game alcoholic spice` - `DERIVED` = `sauce seasoning mix dip syrup filling extract flavored meatless imitation substitute vegetarian vegan giblet liver gizzard pickled puffed` (tokens too: "sauces", "mixes", "giblets" hit them) "Digits" below means a token made only of ASCII `0`–`9`. A "quantity" is digits, or one or more ASCII digits followed by exactly one of the `UNITS` ("2", "100g", "12oz", "2l"; not "2gs" or "1x"). ### 1. Plan the query 1. `q` = normalize(query). 2. `soft` = the tokens of `q` that are quantities or in `SOFT`; `hard` = the other tokens of `q`, in order. If `hard` is empty, `hard` = `q` and `soft` = []. 3. If `hard` has more than 5 tokens, keep the 5 longest (length in Unicode scalars; on a tie the earlier one wins), in their original order. Keep at most the first 5 `soft` tokens. 4. Add to `soft`, after those, every distinct single-scalar piece of the query (step 3 of normalization, before step 4) other than `"s"`, in order of first appearance; these are not capped. ("milk 2%" → hard `milk`, soft `2`; "special k" → hard `special`, soft `k`; "100g chicken breast" → hard `chicken breast`, soft `100g`.) 5. If `hard` is empty, return nothing. ### 2. Fetch and pick candidates 1. Fetch the shard of every `hard` token in parallel (soft tokens are never fetched). `L_t` = `shard.w[t]` (missing means empty). 2. `required` = the `hard` tokens with a non-empty `L_t`, in `hard` order. If there are none, return no online results. 3. Candidates = the records of every `L_t` among `required`, each id once (lists in `hard` order, each in its own order). Keep a candidate only if its `normalize(name) ∪ normalize(brand)` contains every token of `required`. (Lists are capped, so a record can be missing from a long list it belongs in; checking its own tokens finds it from any of them. Over complete lists this is just the records in all of them.) ### 3. Score each candidate With `N` = normalize(name), `Br` = normalize(brand): - `softHit` = how many `soft` tokens are in `P`, where `P` = every piece (any length) of `name + " " + brand` after normalization steps 1–3, plus each piece's stem (step 5). - `segs` = for each part of the raw `name` split on `","`, normalize(part); drop the empty ones. - `neutral` = `NEUTRAL`; `lead` = `N`. If `segs` has 2 or more entries, `segs[0]` is exactly one token, that token is in `CLASS`, it is not in `hard`, and the raw part that gives `segs[1]` does not start with a stopword (its first piece, normalization steps 1–3, is not a stopword after stemming): add it to `neutral`, drop `segs[0]` from `segs`, and set `lead` = the new `segs[0]`. ("Fish, salmon, raw" for "salmon" reads as "salmon, raw"; "Cheese, with wine" stays cheese.) - `core` = the tokens of `N` that are not in `NEUTRAL` (the original list, without the class word), not digits and not in `soft`. - `qcore` = the tokens of `hard` that are not in `neutral`. - `allTokensInName` = every token of `required` is in `N`. - `exactName` = `core` is not empty, `core` equals `qcore` as sets, and `softHit` = the number of `soft` tokens. - `startsWithFirst` = `lead` is not empty and `lead[0]` = `hard[0]`. - `extraNameTokens` = how many tokens of `N` are not in `hard`, not in `neutral`, not digits and not in `soft`. - `brandOnly` = some token of `hard` is in `Br` and not in `N`. - `headSegment` = `segs` is not empty, every token of `segs[0]` is in `hard ∪ neutral`, and at least one is in `hard`. - `derived` = how many tokens of `N` are in `DERIVED` and not in `q`. ``` final = score + 250·allTokensInName + 200·exactName + 80·startsWithFirst − 12·min(extraNameTokens, 10) + 40·headSegment + 40·softHit − 80·derived + 120·(record is generic AND NOT brandOnly) ``` ### 4. Order Deduplicate by id, keeping the higher `final`. Sort by: 1. `final`, highest first; 2. then fewer tokens in `N`; 3. then fewer Unicode scalars in the raw `name`; 4. then `id`, ascending by UTF-8 bytes. Return the top 50. ### 5. Offline and merging The offline bundle is searched with the same plan and the same rules (its postings stand in for `L_t`, uncapped). When the online search succeeded for the same query text, an offline hit is kept only if its `normalize(name) ∪ normalize(brand)` also contains every token of the **online** `required` list; when the online search failed or hasn't run, the offline search stands alone. (Every offline hit has every offline-required token, and every bundle record is in the index, so this keeps all offline hits or none: filtering before or after taking the offline top 50 is the same.) Online and offline hits are merged: one per id (the higher `final`), ordered as in step 4, top 50. ### App as-you-type behaviour (informative; the Python `app` mode mirrors it) **The prefix.** While the text ends inside a word (its last scalar is a letter or number after normalization steps 1–3), and that last piece makes a token that is the last `hard` token, that token is the prefix and the piece itself (after steps 1–3, not stemmed) is the typed piece. So there is none when the word being typed is soft ("milk 12"), was dropped by the 5-longest rule, or repeats an earlier token that isn't the last hard one ("milk tea milk"). (When every token is soft, `hard` = `q` and its last token can still be the prefix: "2 cups".) When `q` is empty and the text ends inside a stopword ("to", "on", "la", "of the"), that piece's stem becomes the only hard token and the prefix; `q` stays empty, so there is no online search. A record **fits** the word being typed when it has the prefix token or a word, as written (pieces of `name + " " + brand` that would be tokens, not stemmed), that starts with the typed piece (by UTF-8 bytes). **Offline**, the prefix token matches the records that fit it. A matched prefix counts towards `required` and `allTokensInName` (when the matching word is in the name) and towards `brandOnly` (when it is only in the brand). Every other term, `derived` included, stays whole-word. **Online**, the search runs on the text, whole words. When there is a prefix, `hard` has 2 or more tokens and no online hit fits the word being typed, it runs again on the finished words (the text's pieces from normalization step 3 without the last one, joined with single spaces), and that search's hits and `required` list are the ones used ("fairlife mi": "mi" is a token of its own, so the full text finds nothing; "fairlife" finds the milks). The online hits are then narrowed to those that fit the word being typed; when none fit, all of them stay (a typo shouldn't hide what the finished words found). **Merging**, the online-`required` filter applies to offline hits as above, except that an offline hit that fits the word being typed is exempt from the prefix token (one that doesn't fit it is not: "diet coke" doesn't keep "Roll, diet"). **App only (no Python model):** online results for an earlier text stay on screen while the search for the new text runs. They are narrowed to the word being typed with no fallback and must still contain every finished hard token, and offline hits must contain each finished hard token that the earlier search required. ## Licensing and attribution - USDA FoodData Central: public domain (CC0 1.0). Credit "USDA, Agricultural Research Service, FoodData Central". - Open Food Facts: Open Database License (ODbL 1.0); individual contents under DbCL 1.0. Attribution is required on every screen that shows OFF data. Share-alike: the derived index is itself published openly at the base URL, and the build scripts are published, which satisfies ODbL 4.6. - Any extra source needs a license that allows redistribution in a commercial app with attribution, and a download that needs no API key or account. Record its exact license and required attribution text in the manifest and on `attribution.html`, plus a `credit` of at most 120 characters that the app shows next to its foods (it must carry whatever the license requires to be shown with the data, e.g. the OGL statement). The app shows `credit` when present, else `attribution` when it is at most 160 characters, else "Food data: "; F, S, N, B and O use fixed app texts.