Repository navigation
perf(hermitcrab): share trailing analysis template slot batches - #546
johnml1135 wants to merge 4 commits into
Conversation
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #546 +/- ##
==========================================
+ Coverage 74.57% 74.69% +0.12%
==========================================
Files 453 454 +1
Lines 38342 38459 +117
Branches 5284 5308 +24
==========================================
+ Hits 28593 28727 +134
+ Misses 8583 8574 -9
+ Partials 1166 1158 -8 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
d6f481e to
5ed55e0
Compare
| if ( | ||
| !_hasSharedSlots | ||
| || _morpher.TraceManager.IsTracing | ||
| || _morpher.MaxDegreeOfParallelism != 1 |
There was a problem hiding this comment.
Minor: F4. A default Morpher never takes the shared walk: maxDegreeOfParallelism defaults to 0 (Morpher.cs:29), so this guard sends every call to base.Apply and only callers who pass degree 1 get fewer batches. The PR body defers parallel sharing but presents the speedup as general; say in the description that it needs degree 1, or say why the default should not be covered.
There was a problem hiding this comment.
How about we do 4 at once? I did a test and it worked better. What do you think? See #582.
Public API, target framework, package, machine.py parity: None verified. Findings: F1 addressed, F2 addressed, F3 addressed, F4 new. Reviewed at 5ed55e0 |
…round AnalysisScope.TemplateMemo no longer wraps the battery on master, so the statement that #546 composes with it is removed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…round AnalysisScope.TemplateMemo no longer wraps the battery on master, so the statement that #546 composes with it is removed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
5ed55e0 to
536686d
Compare
R1, R2, R5 and T1 are kept with their net-of-own-cost numbers in steps, R4 is a recorded dead end, R6 is unbuilt, R3 and R1 on #546 are in progress. Adds ledger rows 38-46. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Rows 43 and 44 move from in progress to done. R1 on #546 is draft PR #577 (commit bbb81a0, shadow 0 violations); R3 ordered look-ahead is draft PR #576 (shadow 0 violations). Adds the round-doc section for both and the PanGloss port issues (sillsdev/PanGloss #65, #66). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…k guard, avoid per-node closures The shared walk is only taken when MaxAlternatives is 0, so its limit check could never fire; the guard comment now names all five fallback conditions; Walk checks admitted templates with a loop instead of a capturing lambda. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#522 removed AnalysisStateKey along with the analysis-cascade memo and kept the same identity as AnalysisStratumRule.AnalysisMergeKey, so the shared-walk equivalence test now compares outputs under that key. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
536686d to
17a6894
Compare
Start here:
src/SIL.Machine.Morphology.HermitCrab/AnalysisAffixTemplatesRule.cs-- the slot trie and the shared walk;AnalysisStratumRulenow builds it instead of a plainRuleBatch.Status: open since 2026-10-22, rebased onto master after #522 (merged). CI green. Review of the design is what it waits on.
FieldWorks consumer: FLEx's parser worker adopts this through
ParseWordson a degree-1 Morpher, tracked in LT-22885.What it does. Templates whose trailing slots are identical (same rule objects, same order, same optionality) now run each shared slot batch once per battery call instead of once per template. Output order and first-seen words per template are unchanged. On a rebased-head benchmark the shared walk cut slot batches by 33.1% (Sena sample) and 21.9% (Mbugwe sample).
The unknown you arrive with: does a shared walk lose or reorder parses? No. Each trie edge is the same ordered rules and optionality beneath the same preceding slots, so every template under it reaches the edge with the word its own walk would. Outputs are still unioned per template, in template order, so the same representatives reach analysis folding. The question worth your time is the guard list, not the walk.
Where to look
SharedWalkMatchesPerTemplateWalkInOrder(4 grammar variants) compares the shared walk with the per-templateRuleBatch, by orderedWordequality and byAnalysisStratumRule.AnalysisMergeKeysequence.UntracedBatteryRunsSharedSlotBatchOncepins two identical tails to one execution, and the next call executing again.Applyguard):TracedAndParallelBatteriesKeepPerTemplateWalk,SharedWalkPreservesBatteryAlternativeLimit,AlternativeLimitStopsBeforeLaterTemplateAdmission,ChangedSlotOptionalityPreservesPerTemplateWalk.AnalysisAffixTemplateRule.CanApplyis now split out so the trie checks each template once.Not here. The shared walk runs only when
Morpher.MaxDegreeOfParallelism == 1. The default degree is 0, which keepsParallelApplySlots, so the speedup applies to degree-1 morphers. FieldWorks HCParser and the HermitCrab tool use the default. Throughput there comes from #582 (open,Morpher.ParseWords), which parses several words at once on the degree-1 path. Also not here: tracing, finiteMaxAlternatives, custom compiled templates, changed slot optionality. Full-corpus parity is not verified.Stack and verification. Independent of the other open HermitCrab PRs; #577 stacks on this. Ran earlier at an older head: HermitCrab test project,
git diff --check, restore and CSharpier. Not run:./local_check.shbuild step (native SentencePiece build needscmake, absent locally; CI builds pass). The conformance suite (#480 head) was not rerun after the rebase; its older results are in Evidence.Next: review the guard list in
AnalysisAffixTemplatesRule.Apply, then approve or name a missing fallback.Reading this a year from now -- start here
This change shares trailing slot batches across affix templates inside one battery call. It is separate from #490's battery-level memo, which #522 removed: that memo cut battery invocations, this cuts repeated slot batches inside one invocation. The precedent is sillsdev/PanGloss ledger entry 095,
docs/divergences/095-shared-template-slot-batches.md, and itsstratum.rsslot trie.Decisions and why
ParallelApplySlots); sharing it needs its own design. Parallelism across words (Add bounded batch parsing to HermitCrab #582) is the planned route for default callers.MaxAlternativesuses the original walk to keep early termination and per-template admission.Parity argument
Recursing through batch outputs in their existing order, then taking the optional unchanged-word continuation, preserves each template's depth-first emission sequence and admits or loses no parses. Each template keeps its own output set, unioned in template order with the existing comparer, so first-seen representatives and ordered words reach deduplication and analysis folding unchanged.
Evidence
Measured at an earlier head, before #522 removed the memo, so absolute batch counts predate the rebase. Single-threaded, sharing disabled versus enabled, new analysis scope per parse, temporary counters only in the output-directory build. The two later figures in the pitch come from a benchmark on the rebased head.
All completed segmentable pairs had identical morpheme-ID/surface signature lists after ordinal sorting, including multiplicity: zero differences, zero errors. Sena had 28 matching invalid-shape outcomes and no capped pairs; Mbugwe excluded 61 pairs that hit a two-second watchdog in either mode. Samples are evenly spaced nonempty rows from each word list. No corpus words, parse examples or grammar contents are included.
Conformance (#480 head, 60 fixtures,
--include-pathological, older head): this branch and the unchanged base both failed 2. The requested 60/60 baseline could not be reproduced: both fail traced-rule attribution inboundary-leftover-in-narrowing-lhs-strrep, anddeep-optional-affix-nestingcrosses its 15-second budget, using the original walk because it has no shared slot tails.Local runs used WSL
~/.dotnet, cached NuGet packages and bounded workers, no network../local_check.shand--agent-strictfailed only at the native SentencePiece build (missingcmake); restore, CSharpier and the comment scan passed.🤖 Generated with Claude Code
This change is