[italomacedo]

Summary

Add cluster mode support to OCL providers (CodeSystem, ValueSet, ConceptMap) following the existing VSAC leader/follower pattern.

How it works

• Master (leader): Discovers sources from OCL, resolves default versions, writes a discovery snapshot (data/terminology-cache/ocl/discovery-snapshot.json) containing full source metadata and CodeSystem JSON. After each discovery/refresh cycle, proactively warms all sources into per-source cold cache files. All writes use atomic temp-file-plus-rename.

• Workers (followers): Read the discovery snapshot to build identical providers and factories (same IDs, caseSensitive, publisher, description, filters, properties as master). Load concepts from per-source cold cache files. Detect snapshot changes on the once-per-minute refresh cycle and create factories for newly discovered sources. Can still fetch individual concepts on demand for cache misses, but never run bulk discovery or background warm-up.

Changes

• tx/ocl/cs-ocl.cjs: Discovery snapshot write/read/deserialize, follower loads from snapshot instead of scanning cold cache, proactive warm-up after discovery, cluster passed explicitly to factory constructor
• tx/ocl/vs-ocl.cjs: Fix follower cache check to use actual params key (not just 'default'), stale tmp cleanup
• tx/ocl/cache/cache-paths.cjs: Add DISCOVERY_SNAPSHOT_PATH
• tx/ocl/cache/cache-utils.cjs: Add cleanupStaleTmpFiles
• tx/library.js: Pass cluster role to factory constructor
• library/role.js: Remove redundant tx.oclCluster (kept only the tx.js derivation)
• cluster.md: Add OCL row to "What runs where" table and OCL paragraph
• tests/ocl/ocl-cluster-follower.test.js: 7 tests for follower initialization, metadata preservation, snapshot reload

Test plan

• All 216 existing OCL tests pass (15 suites)
• 7 new follower tests pass: snapshot loading, metadata preservation, source meta population, id consistency, missing snapshot handling, multiple sources, snapshot reload detection
• Manual test: run with --role=master and verify discovery snapshot is written
• Manual test: run with --role=tx-worker and verify workers serve OCL code systems with correct metadata

🤖 Generated with Claude Code

[grahamegrieve]

Thanks, this is the right direction. The master downloads from OCL and writes the cache files, and the workers read them, the same split we use for VSAC. Making the cache writes atomic is worth having whatever else happens. But I can't merge it yet: as it stands, a worker can't serve any OCL code system's concepts.

Workers get no OCL code system providers. library.loadOcl registers one provider per source, from getSourceMetas(). A follower's new start-up path (#initializeFromColdCache) fills in _codeSystemsByCanonical but never sourceMetaByUrl or _headMetaByCanonical, so getSourceMetas() returns [] and no providers are registered. I checked this on the branch: a follower with a cold-cache file lists the CodeSystem, but has 0 source metas. So on a worker, $lookup, $validate-code and $expand against OCL content fail.

Even with providers, workers would answer differently from the master:

• The CodeSystem is rebuilt from the cache file, which only has url, version and concepts. Name, title, publisher, description, caseSensitive, language and the filter definitions are all lost. caseSensitive changes validation results.
• The id is derived from the url, while the master uses OCL's source id with the provider's prefix. With requests spread across processes, a client can get an id from one and a 404 for it from another.
• A worker only knows the code systems the master has fully downloaded. The master's list of discovered sources is never written anywhere a worker can read it.

Nothing is warmed unless the master is asked. The master only fetches a code system's content when a request on the master misses it. With masterServesTx (the default), the master gets only its share of tx requests, so warming is slow and haphazard. With masterServesTx=false, it never happens.

What I'd suggest:

1. The master writes its whole discovery snapshot (default and HEAD source metas, with the CodeSystem JSON) to one file, using atomicWriteFile, after each discovery or refresh.
2. A worker builds its metas and providers from that file exactly as the master does, so the ids and metadata are the same. It reloads the file when it changes and creates providers for new sources. It loads concepts from the existing per-code-system cache files, as now.
3. The master downloads every source after discovery, rather than waiting for requests.

Smaller things:

• role.js sets tx.oclCluster, but nothing reads it, since tx.js derives it from clusterRole. One of the two can go.
• OCLSourceCodeSystemFactory.cluster is a static that the provider's constructor sets as a side effect. Passing it in explicitly would be clearer.
• A worker's value set check (#followerCheckColdCache) only looks at the default params key. The master keeps one cache file per params key.
• A crash mid-write leaves a .tmp.<pid> file behind. That's minor, but it's worth cleaning these up at startup.
• Please add tests for the follower path, at least: a follower with a cold-cache file registers a provider and can look up a concept from it.
• cluster.md has a table of what each role does, with a VSAC row. It needs an OCL row, and a short paragraph like the VSAC one.

The existing OCL and library tests pass on the branch.

[grahamegrieve]

I'd hoped to approve this one, but it isn't quite ready yet. The snapshot approach fixes the main problem from my last review: workers now have the same providers, ids and metadata as the master. Thanks for working through the rest of the list too. But the rework has introduced some new problems.

1. The snapshot file carries every concept, and is rewritten every minute. Once a code system is complete, #applyConceptsToCodeSystemResource puts its whole concept list into jsonObj.concept. #writeDiscoverySnapshot writes entry.cs.jsonObj as it is, so the snapshot holds every concept of every warm code system. The master writes it on every refresh (getCodeSystemChanges, every minute), whether or not anything changed. Each write changes the file's mtime, so every worker re-reads and re-parses the whole file every minute. Each worker then holds the concepts twice, once from the snapshot and once from the per-source cache file. Please:

• leave concept out of csJson and headCsJson, since the per-source cache files carry the concepts;
• only rewrite the snapshot when its content has changed, for example by comparing a hash of the serialized entries.

2. Proactive warm-up needs a limit. I asked for the master to download every source after discovery. Seeing it implemented, I don't think it can be unconditional:

• It runs on a single-process server too, not only on a cluster's master.
• Discovery prefers the global /sources/ listing, which on a public OCL server is every public organisation-owned source.

Please limit it to cluster mode, or to configured organisations, or make it a setting. Separately, the warm-up in initialize() does nothing. The library creates the factories after listCodeSystems() returns, so factoriesByKey is still empty then, and the first warm-up actually happens on the first refresh.

3. A worker can delete the master's in-progress write. cleanupStaleTmpFiles runs in initialize() before the follower/leader branch, so workers run it too. If a worker starts, or restarts after a memory recycle, while the master is writing, it deletes the master's .tmp.<pid> file and that write fails. Please run the cleanup only on the leader or a single-process server.

4. Workers still don't learn about OCL value sets. Workers skip collection discovery, so they know only the value sets that had a cold-cache file when they started. #followerCheckColdCache only refreshes value sets the worker already knows, so a value set the master caches later never reaches a running worker. Value sets need the same snapshot treatment as code systems: the master writes its collection list, and workers build from it and reload it when it changes.

5. The tests write to the real data folder. The cache paths come from process.cwd(), so running the tests writes into data/terminology-cache/ocl/. After a test run I found a test snapshot there (with the test source http://x.org/cs/PubSrc) and a test cold-cache file. The follower tests also delete discovery-snapshot.json in afterEach. On a working checkout, npm test would overwrite or delete the real snapshot, and a worker running there would pick up the test sources. Please make the cache folder overridable (for example through a setting or environment variable), and have the OCL tests use a temp folder.

For reference: it merges cleanly onto main, the OCL and library tests pass, and lint is clean.

[grahamegrieve]

Thanks Italo — this is much closer. Items 1 and 3 are resolved, and the VS discovery snapshot is the right approach for 4. Remaining:

Test isolation (5): only ocl-cluster-follower.test.js uses a temp OCL_CACHE_DIR. ocl-cs-default-version, ocl-factory-discovery, ocl-discovery-global and ocl-vs-resolve-reference still write discovery-snapshot.json / vs-discovery-snapshot.json (plus codesystems/ and valuesets/) into data/terminology-cache/ocl. A real follower will pick these up. Suggest setting OCL_CACHE_DIR to a temp dir for the whole OCL suite, e.g. in a jest setupFiles for tests/ocl, before any module is required.
Follower VS discovery (4): the follower checks for a newer snapshot only from #scheduleBackgroundExpansion, i.e. when it expands a value set it already knows. A follower that starts before the leader has written the snapshot has an empty map and never looks again. Please also check from searchValueSets / fetchValueSet (cheap: it’s just an mtime check).
Warm-up scope (2): OK to leave as leader-only for now, but please note in cluster.md that the leader warms every source in the global listing.
Please add tests for the follower reading the VS snapshot, and for an unchanged snapshot not being rewritten.

[grahamegrieve]

Thanks Italo, this is very close now. Test isolation, the value set snapshot check on fetch and search, and the note about warm-up scope in cluster.md are all done, and the snapshot changes from the earlier rounds hold up. A last few things, and then I'll merge:

1. The "unchanged snapshot isn't rewritten" test doesn't test that. It calls atomicWriteFile twice, and as its own comment says, that always rewrites. Please drive the leader's own snapshot write twice with the same discovery (with the HTTP client mocked), and check that the file's mtime hasn't changed.
2. fetchValueSetById doesn't check the snapshot. On a worker, GET ValueSet/{id} for a value set the master found after the worker started is not found until something else triggers a reload. It needs the same mtime check as fetchValueSet and searchValueSets.
3. The tests wait with fixed sleeps (setTimeout of 50 and 100 ms) for the background reload. That will be flaky on a loaded CI runner. Please wait on the reload itself (_refreshPromise) instead.
4. Please take package-lock.json out of this PR. The mongoose, saslprep and glob bumps are already on main, so this PR's diff should be just the OCL work.

And for later, not this PR: the OCL cache folder is process.cwd()/data/terminology-cache/ocl, not the data folder. It works in a cluster, because every process has the same working directory. But with FHIRSMITH_DATA_DIR set somewhere else (a global npm install), OCL's cache ends up apart from the rest of the data. Defaulting to folders.filePath('terminology-cache', 'ocl'), and documenting OCL_CACHE_DIR, would fix that.