[jmandel]

Problem

After #379, a LOINC filter checked by $validate-code runs one indexed query per code (lsql). Two filter forms still redo a whole-table read on every request:

• A part given by name (SCALE_TYP = Doc). The membership query was ... TargetKey IN (SELECT CodeKey FROM Codes WHERE Description = 'Doc' COLLATE NOCASE) AND SourceKey = ?. Codes.Description had no index, so each call scanned all 257,267 codes on LOINC 2.83: 17.9 ms CPU for that query alone (plan: LIST SUBQUERY 1 | SCAN Codes). With the two keys written in, the same query takes 0.014 ms.
• A regex (CLASS regex CYTO|HL7\.CYTOGEN|...). #findRegexMatches re-reads the property's values and re-runs the regex on every request.

Change

tx/cs/cs-loinc.js:

• The factory's shared data gets a small LoincFilterLookups that remembers:
  • the CodeKeys for a part name (the same Description = ? COLLATE NOCASE query, run once per name). The key is the name lower-cased as NOCASE compares it (ASCII letters only), so Doc, DOC and doc share one entry;
  • the matching keys for a (property, regex).
• After that, the filter's SQL lists the keys (TargetKey IN (70401,71649)), so the per-code lsql is a plain indexed lookup.
• The lookups are bounded: the 256 most recently used, and at most 1,000,000 keys in all. They hold the keys a name or regex resolves to, not the filter's codes. Heap after GC on the whole workload below was 821 MiB both before and after the change.
• A provider built without the factory's shared data (as tests/cs/sqlite-callback-throws.test.js does) makes its own: this.filterLookups = sharedData.filterLookups || new LoincFilterLookups().
• A name that no code has now gives an empty filter, the way a regex that matches nothing already does. The answer is the same as before.

tx/importers/import-loinc.module.js: a new index next to the other Codes indexes,

CREATE INDEX IF NOT EXISTS idx_codes_description ON Codes(Description COLLATE NOCASE)

The name lookup was the only filter query that read a whole table. On LOINC 2.83 (a copy, index added with the sqlite3 command line, 1.6 s):

	without the index	with it
plan of SELECT CodeKey FROM Codes WHERE Description = ? COLLATE NOCASE	SCAN Codes	SEARCH Codes USING COVERING INDEX idx_codes_description (Description=?)
that query, median of 20 (Doc, DOC, Nar, Qn, a name no code has)	18.1-18.4 ms	0.04 ms
size		14,491,648 bytes (3,538 pages of 4 KB; the file grew from 1,017 MB to 1,032 MB)

The cache stays: a regex can't use an index (every candidate value is tested), and for names it covers databases imported before the index.

CHANGELOG.md (Unreleased): both changes, and the command that adds the index to an existing database without re-importing:

sqlite3 /path/to/loinc.db "CREATE INDEX IF NOT EXISTS idx_codes_description ON Codes(Description COLLATE NOCASE)"

The branch has upstream main (e4abfec) merged in, so the changelog goes in a new Unreleased section above 1.0.2.

Measured

The numbers below are from the first commit (cache only, on a database without the new index). Built-in LOINC 2.83, one server, requests sent one at a time, warm (second pass measured). The numbers are server CPU per call: the machine was shared with other jobs.

Two requests, each sent 100 times (POST /r4/ValueSet/$validate-code):

{"resourceType":"Parameters","parameter":[
 {"name":"coding","valueCoding":{"system":"http://loinc.org","code":"60591-5"}},
 {"name":"valueSet","resource":{"resourceType":"ValueSet","url":"http://example.org/ValueSet/scale-doc","status":"active",
  "compose":{"include":[{"system":"http://loinc.org","filter":[{"property":"SCALE_TYP","op":"=","value":"Doc"}]}]}}}]}

{"resourceType":"Parameters","parameter":[
 {"name":"coding","valueCoding":{"system":"http://loinc.org","code":"2345-7"}},
 {"name":"valueSet","resource":{"resourceType":"ValueSet","url":"http://example.org/ValueSet/lab-not-path","status":"active",
  "compose":{"include":[{"system":"http://loinc.org","filter":[{"property":"STATUS","op":"=","value":"ACTIVE"},{"property":"CLASSTYPE","op":"=","value":"1"}]}],
   "exclude":[{"system":"http://loinc.org","filter":[{"property":"CLASS","op":"regex","value":"CYTO|HL7\\.CYTOGEN|HL7\\.GENETICS|^PATH(\\..*)?|^MOLPATH(\\..*)?|NR STATS|H&P\\.HX\\.LAB|CHALSKIN|LABORDERS"}]}]}}}]}

CPU per call, median	main (482ea83)	this branch
SCALE_TYP = Doc	40.2 ms	1.5 ms
CLASS regex exclude	31.8 ms	2.0 ms

Workload replayed from IG Publisher traffic: 1,866 $validate-code calls against LOINC rule-based value sets, two runs of each, alternating:

	main	this branch
total CPU	14.1 / 13.9 s	5.8 / 5.9 s
total wall	13.0 / 12.7 s	5.0 / 5.1 s
calls over 50 ms	40 / 31	0 / 0
SCALE_TYP = (183 calls), CPU per call	35.3 ms	2.4 ms
CLASS regex + ... (66 calls), CPU per call	20.9 ms	3.2 ms

On the whole replay (44,498 $validate-code calls), CPU went from 120.7 s to 115.1 s, and the LOINC rule-based calls in it from 12.7 s to 6.0 s. All 44,498 answers are identical to main's.

Tests

• tests/cs/cs-loinc.test.js, 2 new tests:
  • SCALE_TYP = Doc and = DOC list and check the codes related to the parts named Doc. A name no part has gives no codes. The "each filter: checked = listed" list now also includes these filters and CLASS regex.
  • "a part name and a regex are looked up once per factory": three requests each (one provider per request, as the server builds them), for the names Doc, DOC and doc. The test counts the statements that read Codes.Description and the CLASS part scan. On main they run [3, 3] times, so the test fails there; with a case-sensitive cache key the names run 3 times. Here they run [1, 1] times.
• tests/cs/sqlite-callback-throws.test.js: 5 of 5 pass (without the constructor fallback, the COMPONENT regex test fails with Cannot read properties of undefined (reading 'get')).
• LOINC suites (cs-loinc, cs-loinc-classtype, sqlite-callback-throws, sqlite-provider-close, cs-worker-readonly, ancestors-of, tx/loinc-import-answer-lists): 116 passed, 2 skipped (7 suites)
• Full tests/cs and tests/tx: 171 suites pass (5 skipped): 3,260 tests pass, 87 skipped (npx jest tests/cs tests/tx, test-cases run separately below)
• tests/tx/test-cases.test.js: 5,251 of 5,251 pass (validator jar via FHIRSMITH_VALIDATOR_JAR)
• eslint is clean on the changed files.

🤖 Generated with Claude Code

[grahamegrieve]

Nice speedup, and the cache looks safe: bounded, held per factory, and nothing is stored when a lookup fails or times out.

One thing before merging: the provider now assumes sharedData.filterLookups is set. tests/cs/sqlite-callback-throws.test.js builds a LoincServices with its own sharedData (no filterLookups) and runs a COMPONENT regex filter. It expects the rejection "Operation exceeded time limit", but this.filterLookups.get will throw a TypeError first. Your test list didn't include that file. Can you fall back in the constructor, this.filterLookups = sharedData.filterLookups || new LoincFilterLookups(), and run the full suite?

Minor: the name key is case-sensitive while the match isn't, so Doc and DOC are scanned and cached separately. Lower-casing the key would save the second scan.

Also, this touches the same #executeFilterQuery as #386. If you take up the comment there about part names in a relationship in, #partKeysByName is what you need: look up each name and use the union of the keys.

[grahamegrieve]

Following up on the name lookup: the real fix is an index. I checked the query plans on the 2.83 database, and Codes WHERE Description = ? COLLATE NOCASE is the only filter query that scans a whole table. Everything else, including the per-code checks and the candidate sets for regexes, already uses an index.

Can you add CREATE INDEX IF NOT EXISTS idx_codes_description ON Codes(Description COLLATE NOCASE) to the importer (tx/importers/import-loinc.module.js, next to the other Codes indexes)? It costs roughly 15–20 MB on a 1 GB database. Existing databases can have it added once with the sqlite3 command line rather than being re-imported. Please also note it in the changelog, so operators know to do that.

Keep the factory cache: regex filters can't use an index, because every candidate value still has to be tested, so caching is the only thing that helps them. For names, the cache still helps databases built before the index, and with the index a miss costs almost nothing anyway.

[jmandel]

Thanks. All three are done (71aa079, on top of a merge of main e4abfec):

• The constructor falls back: this.filterLookups = sharedData.filterLookups || new LoincFilterLookups(). sqlite-callback-throws passes 5/5. Without the fallback it fails just as you said (Cannot read properties of undefined (reading 'get')). The full tests/cs + tests/tx run is in the body.

• The name key is lower-cased the same way NOCASE folds (ASCII letters only), so Doc, DOC and doc share one lookup. The cache test now uses all three names and counts one Description query.

• idx_codes_description ON Codes(Description COLLATE NOCASE) is in the importer, next to the other Codes indexes. On a copy of 2.83 the plan goes from SCAN Codes to SEARCH Codes USING COVERING INDEX idx_codes_description (Description=?), and the lookup from 18 ms to 0.04 ms. The index is 14.5 MB, and adding it with the sqlite3 command line took 1.6 s. The changelog (a new Unreleased section) gives operators the command:

  sqlite3 /path/to/loinc.db "CREATE INDEX IF NOT EXISTS idx_codes_description ON Codes(Description COLLATE NOCASE)"
  

The factory cache stays as it was.

#386 now builds on this branch: a part name in a relationship in uses #partKeysByName.

[grahamegrieve]

Thanks Josh, looks good. I'll tidy up the change log. And it would be good if the server noticed an existing database without idx_codes_description and said so once at startup (or in --check), since operators won’t read the CHANGELOG for a sqlite command; the cache makes a missing index cheap, but the hint costs nothing.

Separately, not this PR: cs-loinc.test.js re-imports data/loinc-testing.db while cs-loinc-classtype, cs-worker-readonly and cluster-worker-readonly copy it, so they can copy a half-written file when run in parallel. cs-loinc-classtype failed that way for me in a combined run on this branch.

I'll merge this, but you might want to follow up on those things.