diff --git a/CHANGELOG.md b/CHANGELOG.md index 4a4eb8b8..ef563724 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,59 @@ All notable changes to PolterType are recorded here. The format is loosely based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and the project follows [Semantic Versioning](https://semver.org/). +## [Unreleased] + +### Added + +- **Arabic.** A bundled `ar` layout (Arabic 101, read off the OS rather + than transcribed), a 2.5 M-form dictionary from Ayaspell's stems + with the article forms added, Arabic's long vowels for the + plausibility detector, and a short + stop-word list checked so that none of its words is what an English + stop word types as. macOS maps `Arabic`, `Arabic - PC`, + `Arabic - QWERTY`, `Arabic - AZERTY` and `Arabic - North Africa` to + it; Linux maps `ara`; Windows maps every country's Arabic keyboard + (`ar-SA`, `ar-EG`, …). A lam-alef key puts two code points on screen, + and a correction erases it as two. +- **macOS reads your keyboards from the OS.** The layout backend now + answers `describe_keymaps` with `UCKeyTranslate`, as the Windows one + does with `ToUnicodeEx`, so a variant whose keys differ from the + bundled table — Apple's own Arabic against Arabic 101, Russian + Phonetic against Russian — is corrected on the keys you actually + have. + +### Fixed + +- **Arabic presentation forms count as Arabic script.** `ﻻ` and the + rest of U+FB50–FDFF and U+FE70–FEFF read as "other script" and + dragged a word's script fit down. +- **`cargo xtask wordlists fetch` finds a `SET` anywhere in the `.aff`, + tab-separated or not,** and no longer parses rules it will not use + for a stems-only dictionary. + +- **Hyprland and sway knew only six of the bundled languages by name.** + Those compositors report a keymap's description, and `Polish`, + `Czech`, `Greek`, `Hebrew`, `Turkish`, `Bulgarian`, `Italian`, + `Portuguese` and `Arabic` came back as unknown layouts. +- **A correctly typed Hebrew word is no longer replaced on shape alone.** + Hebrew writes no short vowels, so a word missing from its stems list + scored as one consonant cluster and could lose to its Latin rendering. + Leaving Hebrew or Arabic is now the dictionaries' call: a word in neither + moves only when the other layout's dictionary follows much more of it + than its own, so `microsoft` comes back and `בישראל` stays. On Wikipedia + text about 40 % of out-of-dictionary English returns, and none of the real + words the shape check used to replace is touched by this rule. +- **On Windows, switching between two keyboards of one language now lands + on the keyboard whose table PolterType holds,** rather than on whichever + the OS listed last. +- **Converting a selection now plays the correction chime.** With + `[selection] enabled`, pressing the force-switch hotkey on a selected + passage converted it, pasted it back and put the clipboard back in + complete silence — the chime lived only in the word paths, so a press + that worked looked like one that did nothing. The selection path now + chimes too, on the same "Play a soft chime on correction" setting as + every other path, and stays silent when there was nothing to convert. + ## [0.38.2] — hands off first ### Fixed diff --git a/README.md b/README.md index b98ab22e..8926aac6 100644 --- a/README.md +++ b/README.md @@ -18,7 +18,7 @@ typed on an English layout comes out as `ma;ana` — PolterType fixes the word the moment it ends and switches the layout with it, so `por la tarde` lands correctly as typed.* -> **Status: v0.38.2**, out of beta since v0.1.0. Windows, macOS and +> **Status: v0.39.0**, out of beta since v0.1.0. Windows, macOS and > Linux (Wayland and X11, on the desktops in the table below) all > correct words end to end, and the spelling-suggestions tooltip > renders on all three. Installers are **unsigned**, so first launch @@ -43,7 +43,7 @@ hotkeys, smart commands, wordlists, and per-app exceptions.* - **Fast** — pure Rust, no WebView, no perceptible typing latency. - **Light to run** — single binary, ~20 MB, a tray icon's worth of CPU and RAM. **The download is not light** — the installers are - about 60–70 MB, nearly all of it the fifteen bundled dictionaries — but + about 60–70 MB, nearly all of it the sixteen bundled dictionaries — but only the languages your OS has enabled are ever read into memory. - **Quiet** — tray-only, **zero telemetry**. Exactly one network call exists, the update check (§ [Staying up to @@ -386,24 +386,24 @@ them on the **Commands** pane in Settings. ## Languages -Fifteen layouts ship with the app, each with a full dictionary: +Sixteen layouts ship with the app, each with a full dictionary: **English (US) · Ukrainian · Russian · German · Spanish · French · Polish · Czech · Greek · Hebrew · Turkish · Bulgarian · Italian · -Portuguese (PT) · Portuguese (BR)** +Portuguese (PT) · Portuguese (BR) · Arabic** PolterType only loads the ones your OS actually has enabled, so -bundling fifteen costs a two-keyboard user nothing at runtime. +bundling sixteen costs a two-keyboard user nothing at runtime. -**On Windows the bundled table is only a starting point.** A Windows -layout is named after a *language*, but a language is not a keyboard — +**On Windows and macOS the bundled table is only a starting point.** A +layout there is named after a *language*, but a language is not a keyboard — Bulgarian alone ships in three genuinely different arrangements. Since v0.14.3 PolterType asks Windows what each of your installed keyboards actually produces and uses that answer, so variants and even custom -layouts we have never heard of come out right. Linux and macOS read -the bundled tables, and a TOML of your own in -`/poltertype/layouts/` still outranks everything, -everywhere. +layouts we have never heard of come out right; macOS does the same +through `UCKeyTranslate`. Linux reads the bundled tables, and a TOML of +your own in `/poltertype/layouts/` still outranks +everything, everywhere. Two are worth a footnote rather than a surprise: @@ -418,6 +418,11 @@ Two are worth a footnote rather than a surprise: because expanding its clitic prefixes yields 60 million of them. Hebrew shares its script with nothing else bundled, so detection leans on that and the dictionary refines it. +- **Arabic** follows Hebrew for the same reason — its article and + clitic particles are prefixes — and adds the article forms (`ال`, + `بال`, `لل`, …) back to the list. The lam-alef keys (`لا` on `B`) type + two code points, and a key here is one character, so they carry the + single-code-point form `ﻻ`, which looks the same. **Yours isn't here?** Adding a language is one TOML file and one wordlist — no Rust required. The walkthrough is diff --git a/crates/poltertype-core/build.rs b/crates/poltertype-core/build.rs index 0615c07d..5449aa5b 100644 --- a/crates/poltertype-core/build.rs +++ b/crates/poltertype-core/build.rs @@ -42,6 +42,7 @@ const LAYOUTS: &[(&str, &str)] = &[ ("it_it", "it-IT"), ("pt_pt", "pt-PT"), ("pt_br", "pt-BR"), + ("ar", "ar"), ]; fn main() { diff --git a/crates/poltertype-core/src/engine/switcher/correction.rs b/crates/poltertype-core/src/engine/switcher/correction.rs index 409fb6aa..4cc99ad5 100644 --- a/crates/poltertype-core/src/engine/switcher/correction.rs +++ b/crates/poltertype-core/src/engine/switcher/correction.rs @@ -256,6 +256,9 @@ impl SwitcherEngine { let mut live = live; let mut click_allowance = pointer_click_allowance; let mut tail: Vec = Vec::new(); + // What the keys in `tail` cost beyond one Backspace each, counted + // under the layout in effect when each landed. + let mut tail_extra = 0usize; let mut resume: Option = None; let mut suspicious = false; @@ -276,6 +279,7 @@ impl SwitcherEngine { && Instant::now() < deadline { let w = self.drain_correction_window(rx, &mut click_allowance); + tail_extra += self.extra_for_events(from, &w.word_keys); tail.extend(w.word_keys); suspicious |= w.suspicious; resume = w.resume; @@ -350,6 +354,7 @@ impl SwitcherEngine { let mut quiet_probes = 0u8; loop { let w = self.drain_correction_window(rx, &mut click_allowance); + tail_extra += self.extra_for_events(to, &w.word_keys); tail.extend(w.word_keys); suspicious |= w.suspicious; if let Some(r) = w.resume { @@ -450,7 +455,7 @@ impl SwitcherEngine { let mut held = HeldKeys::acquire(&self.key_gate); let mut repairs_left = INTRUSION_REPAIRS; - let mut to_delete = backspaces + tail.len() + usize::from(resume.is_some()); + let mut to_delete = backspaces + tail.len() + tail_extra + usize::from(resume.is_some()); loop { // ── Delete: word + boundary + absorbed tail ───────────── // @@ -474,7 +479,7 @@ impl SwitcherEngine { std::thread::sleep(POST_EMIT_LAG); let w = self.drain_correction_window(rx, &mut click_allowance); suspicious |= w.suspicious; - let mut extra = w.word_keys.len(); + let mut extra = w.word_keys.len() + self.extra_for_events(to, &w.word_keys); tail.extend(w.word_keys); if let Some(r) = w.resume { if is_submission_scancode(r.scancode) || resume.is_some() { @@ -515,7 +520,8 @@ impl SwitcherEngine { Some(rk) => { let mut full: Vec = rk.to_vec(); full.extend(extra_keys.iter().copied()); - emitted = full.len(); + emitted = full.len() + + self.extra_backspaces(to, full.iter().map(|k| (k.scancode, k.shift))); let sent = self.key_emitter.send_keys(&full); self.push_echoes(self.key_emitter.take_emitted()); match sent { @@ -682,7 +688,7 @@ impl SwitcherEngine { let w = self.drain_correction_window(rx, &mut click_allowance); let saw_press = w.saw_user_press; suspicious |= w.suspicious; - intruders += w.word_keys.len(); + intruders += w.word_keys.len() + self.extra_for_events(to, &w.word_keys); tail.extend(w.word_keys); if let Some(r) = w.resume { if is_submission_scancode(r.scancode) || resume.is_some() { @@ -999,6 +1005,11 @@ impl SwitcherEngine { std::thread::sleep(PASTE_SETTLE); self.restore_clipboard(clipboard, previous.as_deref()); + // Text was just replaced on screen as in `apply_correction`; with no + // chime here a press that worked looked like one that did nothing. + if self.settings.snapshot().general.sound_on_correct { + self.audio.play(SoundEvent::Correct); + } info!(%from, %to, "selection converted"); let _ = self.out_tx.send(SwitcherEvent::Corrected { from_layout: from, @@ -1012,6 +1023,22 @@ impl SwitcherEngine { } impl SwitcherEngine { + /// What `keys` cost in Backspaces beyond one each, typed under `layout`. + pub(super) fn extra_backspaces( + &self, + layout: &LayoutId, + keys: impl IntoIterator, + ) -> usize { + self.layouts + .get(layout) + .map_or(0, |m| m.extra_backspaces(keys)) + } + + /// [`Self::extra_backspaces`] for keystrokes off the live stream. + fn extra_for_events(&self, layout: &LayoutId, keys: &[KeyEvent]) -> usize { + self.extra_backspaces(layout, keys.iter().map(|k| (k.scancode, k.modifiers.shift))) + } + /// Wait for the finger to come off the hotkey before anything is /// emitted — the same fact `apply_correction` waits on, on the one /// force-switch path that was not waiting on it. @@ -1167,7 +1194,7 @@ impl SwitcherEngine { /// The order has to be the same in every process. `HashMap::keys()` /// was not, which is harmless while the DB holds exactly the two /// active layouts and not at all harmless when it does not: a - /// failed `list_active()` loads all fifteen bundled layouts, the + /// failed `list_active()` loads every bundled layout, the /// force-switch aims at whichever came out first, and the /// pre-flight refuses it — no keystroke, no word, no explanation. /// So: what the OS says is switchable, else the DB sorted by name. @@ -1259,7 +1286,8 @@ impl SwitcherEngine { // The word plus, if there is one, the boundary key that closed // it. A word still being typed has none: nothing follows the // caret to backspace over or put back. - let mut backspaces = last.keys.len(); + let mut backspaces = last.keys.len() + + self.extra_backspaces(&from, last.keys.iter().map(|k| (k.scancode, k.shift))); if let Some(b) = last.boundary { corrected.push(b.ch); // Enter/Tab excepted, where a re-press would submit the line diff --git a/crates/poltertype-core/src/engine/switcher/decide.rs b/crates/poltertype-core/src/engine/switcher/decide.rs index e1ddd889..2b476d32 100644 --- a/crates/poltertype-core/src/engine/switcher/decide.rs +++ b/crates/poltertype-core/src/engine/switcher/decide.rs @@ -268,7 +268,10 @@ impl SwitcherEngine { self.dismiss_suggestions(None); // Counting keys rather than rendered chars survives // scancodes the mapping table cannot render. - self.dispatch_smart_command(cmd, erase_len(cmd, keys.len()), boundary_char); + let typed = keys.len() + + self + .extra_backspaces(¤t_layout, keys.iter().map(|k| (k.scancode, k.shift))); + self.dispatch_smart_command(cmd, erase_len(cmd, typed), boundary_char); // The trigger text is gone from screen: not re-openable by // backspace, not matchable again. buffer.forget_completed(); @@ -470,7 +473,12 @@ impl SwitcherEngine { corrected_text: corrected_with_boundary, // Keys, not rendered chars: under-counting is how // word heads get left behind. - backspaces: keys.len() + 1, + backspaces: keys.len() + + self.extra_backspaces( + ¤t_layout, + keys.iter().map(|k| (k.scancode, k.shift)), + ) + + 1, reason: v.reason, } } diff --git a/crates/poltertype-core/src/engine/switcher/suggestions.rs b/crates/poltertype-core/src/engine/switcher/suggestions.rs index 538fd3a1..eca27c16 100644 --- a/crates/poltertype-core/src/engine/switcher/suggestions.rs +++ b/crates/poltertype-core/src/engine/switcher/suggestions.rs @@ -407,7 +407,16 @@ impl SwitcherEngine { // delete all of it, retype with the word replaced. The accept // chord's digit is not on screen: every chord holds Ctrl, Alt or // Meta, and the engine reads such a keystroke as a shortcut. - let backspaces = pending.keys.len() + pending.trail.len(); + let backspaces = pending.keys.len() + + pending.trail.len() + + self.extra_backspaces( + &pending.layout, + pending + .keys + .iter() + .chain(pending.trail.iter().map(|t| &t.key)) + .map(|k| (k.scancode, k.shift)), + ); // Cross-layout entries replay the original scancodes under the // switched layout; spelling entries reverse-map the suggestion diff --git a/crates/poltertype-core/src/engine/tests.rs b/crates/poltertype-core/src/engine/tests.rs index 0a35db97..0f35e612 100644 --- a/crates/poltertype-core/src/engine/tests.rs +++ b/crates/poltertype-core/src/engine/tests.rs @@ -372,6 +372,7 @@ mod engine_integration_tests { // The gate is a no-op in most tests: they exercise the // path taken when keystrokes cannot be held back. poltertype_input::KeyGate::disabled(), + ["en-US", "uk-UA"], ) } @@ -388,6 +389,27 @@ mod engine_integration_tests { tweak, None, poltertype_input::KeyGate::claiming_without_listener(), + ["en-US", "uk-UA"], + ) + } + + /// en-US and Arabic active, with a detector that always wants the + /// other one: what a lam-alef key costs is the whole question. + fn start_arabic() -> Self { + Self::start_gated( + 60_000, + MockEmitter::default(), + false, + None, + Some(vec![Box::new(AlwaysOther( + LayoutId::from("en-US"), + LayoutId::from("ar"), + ))]), + None, + |_| {}, + None, + poltertype_input::KeyGate::disabled(), + ["en-US", "ar"], ) } @@ -402,6 +424,7 @@ mod engine_integration_tests { tweak: impl FnOnce(&mut crate::settings::Settings), clipboard: Option>, key_gate: poltertype_input::KeyGate, + active_layouts: [&str; 2], ) -> Self { let mut settings = crate::settings::Settings::default(); settings.engine.idle_timeout_ms = idle_timeout_ms; @@ -416,7 +439,7 @@ mod engine_integration_tests { // which keys count as letters. bg-BG puts `б` on the `/` // key, which quietly made `/tmp` one four-key token here // while a real en-US + uk-UA machine sees a path. - let active = [LayoutId::from("en-US"), LayoutId::from("uk-UA")]; + let active = active_layouts.map(LayoutId::from); let layouts = Arc::new( LayoutDb::load(crate::layouts::LoadOptions { active_filter: Some(&active), @@ -425,7 +448,7 @@ mod engine_integration_tests { .expect("bundled layouts load"), ); let emitter = Arc::new(emitter); - let mut switcher = MockSwitcher::new("en-US", &["en-US", "uk-UA"]); + let mut switcher = MockSwitcher::new(active_layouts[0], &active_layouts); switcher.fail_switch = fail_switch; let switcher = Arc::new(switcher); let detectors: Vec> = detectors_override.unwrap_or_else(|| { @@ -754,6 +777,157 @@ mod engine_integration_tests { ); } + /// Both directions of the Arabic ↔ English mix-up through the real + /// pipeline, and neither word disturbed in its own layout. Renders + /// each word through both tables the way the engine does, then asks + /// the detectors. + #[test] + fn arabic_and_english_typed_in_the_wrong_layout_are_switched() { + use crate::layouts::LayoutDb; + let db = LayoutDb::load_embedded(); + let (en, ar) = (LayoutId::from("en-US"), LayoutId::from("ar")); + let en_map = db.get(&en).expect("en-US"); + let ar_map = db.get(&ar).expect("ar"); + let render = |typed: &str| -> String { + typed + .chars() + .map(|ch| { + let (sc, shift) = en_us_key(en_map, ch); + let (plain, shifted) = ar_map.keys[&sc]; + if shift { + shifted.unwrap_or(plain) + } else { + plain + } + }) + .collect() + }; + let judge = |current: &LayoutId, typed: &str| -> Option { + let candidates = vec![(en.clone(), typed.to_owned()), (ar.clone(), render(typed))]; + let ctx = poltertype_detect::DetectionContext { + current_layout: current, + candidates: &candidates, + recent_context: "", + }; + for d in real_detectors() { + match d.judge(&ctx) { + Verdict::NoOpinion => continue, + Verdict::Keep { .. } => return None, + Verdict::Switch(v) => return Some(v.best_layout), + } + } + None + }; + // السلام عليكم, كتاب, مدرسة on Arabic 101 — and السلام again + // with its lam-alef typed on `B`, which renders as `ﻻ`. + for typed in ["hgsghl", "ugd;l", ";jhf", "l]vsm", "hgsbl"] { + assert_eq!( + judge(&en, typed), + Some(ar.clone()), + "{typed:?} -> {}", + render(typed) + ); + assert_eq!( + judge(&ar, typed), + None, + "{} is Arabic typed in Arabic", + render(typed) + ); + } + // بسرعه and كتب carry no long vowel: one consonant cluster to the + // shape score, while their en-US renderings look like words. + // Typed in Arabic they must stay Arabic. + for typed in ["fsvui", ";jf"] { + assert_eq!( + judge(&ar, typed), + None, + "{} is Arabic typed in Arabic", + render(typed) + ); + } + for typed in ["hello", "world", "table", "school"] { + assert_eq!(judge(&en, typed), None, "{typed:?} is English"); + assert_eq!( + judge(&ar, typed), + Some(en.clone()), + "{typed:?} typed in Arabic" + ); + } + } + + /// What real detectors make of `typed` on an abjad layout's keys. + fn judge_on_abjad(abjad: &str, typed: &str) -> Option { + use crate::layouts::LayoutDb; + let db = LayoutDb::load_embedded(); + let (en, other) = (LayoutId::from("en-US"), LayoutId::from(abjad)); + let en_map = db.get(&en).expect("en-US"); + let other_map = db.get(&other).expect("abjad"); + let rendered = en_map.transliterate_to(typed, other_map).expect("renders"); + // Selection paste expands lam-alef; a word in the buffer holds the + // key's one code point. + let rendered = rendered.replace("\u{0644}\u{0627}", "\u{FEFB}"); + let candidates = vec![(en.clone(), typed.to_owned()), (other.clone(), rendered)]; + let ctx = poltertype_detect::DetectionContext { + current_layout: &other, + candidates: &candidates, + recent_context: "", + }; + for d in real_detectors() { + match d.judge(&ctx) { + Verdict::NoOpinion => continue, + Verdict::Keep { .. } => return None, + Verdict::Switch(v) => return Some(v.best_layout), + } + } + None + } + + /// English the dictionary lacks, typed on an Arabic or Hebrew keyboard, + /// comes back; gibberish does not, since no dictionary follows it. + #[test] + fn english_outside_the_dictionary_leaves_an_abjad_on_its_stem() { + let en = LayoutId::from("en-US"); + for abjad in ["ar", "he-IL"] { + for typed in ["hello", "microsoft", "macos", "posix", "distros"] { + assert_eq!( + judge_on_abjad(abjad, typed), + Some(en.clone()), + "{abjad}: {typed}" + ); + } + assert_eq!(judge_on_abjad(abjad, "qwrtpz"), None, "{abjad}: gibberish"); + } + } + + /// The commonest words of Wikipedia text that the stems lists lack, in + /// their own layouts: inflected, or behind a clitic. Shape alone turned + /// a fifth of running Hebrew into Latin (DECISIONS.md, 2026-11-04). + #[test] + fn real_words_outside_the_dictionary_stay_in_their_abjad() { + let db = crate::layouts::LayoutDb::load_embedded(); + let en_map = db.get(&LayoutId::from("en-US")).expect("en-US"); + for (abjad, words) in [ + ( + "ar", + "العربية المصرية المسلمين الإسلامية الشمسية اللغات الرياضية الآلات عربية مختلفة الرئيسية العالمية الرسمية التاريخية البيانات باستعمال العلمية معينة", + ), + ( + "he-IL", + "בישראל בעיר המאה בירושלים המדינה ההיסטוריה בעיקר המחשב הספר בעולם כאשר במהלך המשחק באופן העולם הכדורגל הראשון כיום השנים במקום הרפואה", + ), + ] { + let layout = db.get(&LayoutId::from(abjad)).expect("abjad"); + for word in words.split_whitespace() { + let typed = layout.transliterate_to(word, en_map).expect("renders"); + assert_eq!( + judge_on_abjad(abjad, &typed), + None, + "{abjad}: {word} ({typed})" + ); + } + } + } + /// Issue #33, the residue on 0.21.0: `auto-switch ` typed in /// en-US came back as `ФГЕЩ-ЫЦШЕСР `. Both halves of that need /// explaining, and this pins the half that needs no Caps Lock — @@ -996,7 +1170,7 @@ mod engine_integration_tests { /// by scancode against the *new* layout. /// /// The reported key was `Shift`+`0x35`, but this harness loads all - /// fifteen bundled layouts and bg-BG carries a letter there, which + /// bundled layouts and bg-BG carries a letter there, which /// makes it a word key rather than a boundary. Hence the same trap /// one row up: `Shift`+`0x08` is `?` under uk-UA and `&` under /// en-US, and `?` lives on `Shift`+`0x35` in en-US. @@ -1421,6 +1595,51 @@ mod engine_integration_tests { ); } + /// `B` on Arabic is `لا`: two code points for one key. Typed as part of + /// the word, or raced in behind it, it costs a second Backspace. + #[test] + fn a_lam_alef_key_is_erased_as_two_code_points() { + const B: u32 = 0x30; + let h = Harness::start_arabic(); + *h.switcher.current.lock() = LayoutId::from("ar"); + type_word(&h, &[0x14, B, 0x22]); + h.tap(SPACE); + h.settle(); + let (ops, _) = h.stop(); + assert_eq!( + ops[0], + EmitOp::Backspaces(5), + "3 keys, 1 pair, 1 space: {ops:?}" + ); + + let h = Harness::start_arabic(); + type_word(&h, &[0x14, 0x22]); + h.tap(SPACE); + h.press(B); + h.release(B); + h.settle(); + let (ops, _) = h.stop(); + assert_eq!( + ops[0], + EmitOp::Backspaces(5), + "a raced `B` landed under the Arabic it was switched to: {ops:?}" + ); + + let h = Harness::start_arabic(); + *h.switcher.current.lock() = LayoutId::from("ar"); + type_word(&h, &[0x14, 0x22]); + h.tap(SPACE); + h.press(B); + h.release(B); + h.settle(); + let (ops, _) = h.stop(); + assert_eq!( + ops[0], + EmitOp::Backspaces(4), + "a raced `B` landed under the English it was switched to: {ops:?}" + ); + } + /// The full fast-typing race: the user types the second word and its /// boundary before the first correction begins. Everything must come /// out in order, and word2 must get its own decision. @@ -1705,8 +1924,17 @@ mod engine_integration_tests { /// clipboard to fall back to, with the focused application's half /// of a copy wired to the emitter: press copy, and `selection` /// appears on the clipboard, exactly as an application answers - /// `Ctrl+C`. + /// `Ctrl+C`. Default settings in every other respect. fn selection_harness(selection: &str) -> (Harness, Arc) { + selection_harness_configured(selection, |_| {}) + } + + /// The wide harness: `tweak` gets the whole `Settings` before the + /// engine starts; this harness always enables `[selection]`. + fn selection_harness_configured( + selection: &str, + tweak: impl FnOnce(&mut crate::settings::Settings), + ) -> (Harness, Arc) { let clipboard = Arc::new(MockClipboard::default()); let emitter = MockEmitter::default(); let cb = Arc::clone(&clipboard); @@ -1724,7 +1952,10 @@ mod engine_integration_tests { None, None, None, - |s| s.selection.enabled = true, + |s| { + s.selection.enabled = true; + tweak(s); + }, Some(Arc::clone(&clipboard) as Arc), ); (h, clipboard) @@ -1759,6 +1990,31 @@ mod engine_integration_tests { Some("something the user copied earlier"), "and the user's own clipboard must be the last thing put back: {writes:?}" ); + assert!( + matches!(h.sounds().as_slice(), [crate::audio::SoundEvent::Correct]), + "and a converted selection must chime, once, like every other correction" + ); + } + + /// Issue #47's lesson read on the selection path too: the chime + /// gate must hold there as well. Chime off still converts — it + /// just says nothing. + #[test] + fn a_converted_selection_is_silent_when_the_chime_is_off() { + let (h, clipboard) = + selection_harness_configured("ghbdsn", |s| s.general.sound_on_correct = false); + h.cmd_tx + .send(EngineCommand::SwitchLastForcefully) + .expect("engine alive"); + h.settle(); + + assert!( + clipboard.writes().iter().any(|w| w == "привіт"), + "the conversion must still happen: {:?}", + clipboard.writes() + ); + let sounds = h.sounds(); + assert!(sounds.is_empty(), "chime off means no chime: {sounds:?}"); } /// Issue #51, the half that only X11 shows: while the chord that diff --git a/crates/poltertype-core/src/layouts/consts.rs b/crates/poltertype-core/src/layouts/consts.rs index ce3f8190..fff9862b 100644 --- a/crates/poltertype-core/src/layouts/consts.rs +++ b/crates/poltertype-core/src/layouts/consts.rs @@ -11,7 +11,7 @@ /// "missing TOML" warning at startup, never silently. pub const BUNDLED_LAYOUT_STEMS: &[&str] = &[ "en_us", "uk_ua", "ru_ru", "de_de", "es_es", "fr_fr", "pl_pl", "cs_cz", "el_gr", "he_il", - "tr_tr", "bg_bg", "it_it", "pt_pt", "pt_br", + "tr_tr", "bg_bg", "it_it", "pt_pt", "pt_br", "ar", ]; /// Fewest keys an [`poltertype_types::OsKeymap`] must carry before it diff --git a/crates/poltertype-core/src/layouts/helpers.rs b/crates/poltertype-core/src/layouts/helpers.rs index 7028a863..63fb59b5 100644 --- a/crates/poltertype-core/src/layouts/helpers.rs +++ b/crates/poltertype-core/src/layouts/helpers.rs @@ -46,6 +46,11 @@ pub fn derive_vowels(id: &LayoutId, script: Script) -> Vec { "bg-BG" => "аеиоуъюя".chars().collect(), "it-IT" => "aeiouàèéìíîòóùú".chars().collect(), "pt-PT" | "pt-BR" => "aeiouáàâãéêíóôõúü".chars().collect(), + // Arabic writes only its long vowels: alef in every hamza and + // madda form, waw, yeh, alef maksura, and the lam-alef keys. + // An empty set scored every Arabic word as one long consonant + // cluster — 0.0 — so plausibility could never pick Arabic. + "ar" => "اأإآويىﻻﻵﻷﻹ".chars().collect(), _ => match script { Script::Latin => "aeiouy".chars().collect(), Script::Cyrillic => "аеиіоуюяєїыэё".chars().collect(), diff --git a/crates/poltertype-core/src/layouts/tests.rs b/crates/poltertype-core/src/layouts/tests.rs index 46cc47fb..46b6df62 100644 --- a/crates/poltertype-core/src/layouts/tests.rs +++ b/crates/poltertype-core/src/layouts/tests.rs @@ -12,7 +12,7 @@ fn embedded_layouts_load() { let db = LayoutDb::load_embedded(); for id in [ "en-US", "uk-UA", "ru-RU", "de-DE", "es-ES", "fr-FR", "pl-PL", "cs-CZ", "el-GR", "he-IL", - "tr-TR", "bg-BG", "it-IT", "pt-PT", "pt-BR", + "tr-TR", "bg-BG", "it-IT", "pt-PT", "pt-BR", "ar", ] { assert!( db.get(&LayoutId::from(id)).is_some(), @@ -112,6 +112,9 @@ fn new_languages_translate_distinctive_keys() { // Hebrew is unicase — the letter positions are RTL glyphs. ("he-IL", 0x1E, false, 'ש'), ("he-IL", 0x14, false, 'א'), + // Arabic is unicase too; `B` types lam-alef as one code point. + ("ar", 0x1E, false, 'ش'), + ("ar", 0x30, false, '\u{FEFB}'), // Turkish's dotless/dotted i pair, the one that bites. ("tr-TR", 0x17, false, 'ı'), ("tr-TR", 0x17, true, 'I'), @@ -233,6 +236,7 @@ fn every_bundled_dictionary_knows_its_own_words() { ("cs-CZ", &["ahoj", "slovo", "dobrý", "protože", "děkuji"]), ("el-GR", &["λέξη", "είναι", "καλημέρα", "ευχαριστώ"]), ("he-IL", &["שלום", "מילה", "ספר", "תודה"]), + ("ar", &["كتاب", "مدرسة", "شكرا", "جميل"]), ( "tr-TR", &["kitap", "merhaba", "kelime", "güzel", "teşekkür"], @@ -1127,3 +1131,47 @@ fn a_file_changed_on_disk_is_read_again() { dir.write("words.fst", "newer"); assert_eq!(fst_cache::leaked_bytes(&path).expect("read"), b"newer"); } + +// ─── Lam-alef: one key, two code points ─────────────────────────── + +#[test] +fn a_lam_alef_key_costs_a_second_backspace() { + const B: u32 = 0x30; + const G: u32 = 0x22; + let db = LayoutDb::load_embedded(); + let ar = db.get(&LayoutId::from("ar")).expect("ar is bundled"); + assert_eq!(ar.extra_backspaces([(G, false), (0x23, false)]), 0); + assert_eq!(ar.extra_backspaces([(G, false), (B, false)]), 1); + assert_eq!(ar.extra_backspaces([(B, true)]), 1, "the shifted form too"); + + let en = db.get(&LayoutId::from("en-US")).expect("en-US is bundled"); + assert_eq!(en.extra_backspaces([(B, false)]), 0); +} + +#[test] +fn a_converted_selection_is_pasted_as_the_keyboard_types_it() { + let db = LayoutDb::load_embedded(); + let en = db.get(&LayoutId::from("en-US")).expect("en-US is bundled"); + let ar = db.get(&LayoutId::from("ar")).expect("ar is bundled"); + assert_eq!( + en.transliterate_to("b", ar).as_deref(), + Some("\u{0644}\u{0627}") + ); +} + +#[test] +fn a_lam_alef_pair_converts_back_to_its_one_key() { + let db = LayoutDb::load_embedded(); + let en = db.get(&LayoutId::from("en-US")).expect("en-US is bundled"); + let ar = db.get(&LayoutId::from("ar")).expect("ar is bundled"); + let there = en.transliterate_to("table", ar).expect("converts"); + assert_eq!(ar.transliterate_to(&there, en).as_deref(), Some("table")); + assert_eq!(ar.transliterate_to("\u{FEFB}", en).as_deref(), Some("b")); +} + +#[test] +fn bundled_arabic_digits_are_what_its_keyboard_types() { + let db = LayoutDb::load_embedded(); + let ar = db.get(&LayoutId::from("ar")).expect("ar is bundled"); + assert_eq!(ar.translate_key(plain(0x02)), Some('1')); +} diff --git a/crates/poltertype-core/src/layouts/types.rs b/crates/poltertype-core/src/layouts/types.rs index 82cb4c5d..aae30f2d 100644 --- a/crates/poltertype-core/src/layouts/types.rs +++ b/crates/poltertype-core/src/layouts/types.rs @@ -5,7 +5,7 @@ use std::collections::HashMap; use std::path::Path; use poltertype_detect::{LayoutDictionary, LayoutProfile, Script}; -use poltertype_types::{LayoutId, WordKey}; +use poltertype_types::{LayoutId, WordKey, fold_lam_alef, lam_alef_pair}; use serde::Deserialize; use tracing::warn; @@ -118,6 +118,23 @@ impl LayoutMapping { } } + /// How many of these keystrokes put two code points on screen, each + /// taking a second Backspace: a text field deletes `لا` a code point at + /// a time, and a count of keys alone leaves a word's head behind. + pub fn extra_backspaces(&self, keys: impl IntoIterator) -> usize { + keys.into_iter() + .filter(|&(scancode, shift)| { + self.translate_key(WordKey { + scancode, + shift, + caps: false, + timestamp_ms: 0, + }) + .is_some_and(|c| lam_alef_pair(c).is_some()) + }) + .count() + } + /// Reverse lookup: which physical key and shift state produces `ch` /// under this layout? Used by the suggestion-accept path to type a /// replacement as scancodes. Linear over the ~48-key table, a @@ -208,7 +225,11 @@ impl LayoutMapping { } let mut out = String::with_capacity(text.len()); let mut changed = false; - for ch in text.chars() { + // Where a key types `لا` whole, that pair came off that key: read + // apart it would turn `b` into `gh`. A pair typed on the two keys + // reads the same on screen and loses here; it is the rarer one. + let folded = self.fold_pairs(text); + for ch in folded.chars() { let mapped = self.key_for_char(ch).and_then(|(scancode, shift)| { to.translate_key(WordKey { scancode, @@ -223,7 +244,12 @@ impl LayoutMapping { match mapped { Some(c) => { changed |= c != ch; - out.push(c); + // Pasted text should read as the keyboard would have + // typed it, not as the form our table stands in with. + match lam_alef_pair(c) { + Some(pair) => out.extend(pair), + None => out.push(c), + } } None => out.push(ch), } @@ -231,6 +257,26 @@ impl LayoutMapping { changed.then_some(out) } + /// `text` with each lam-alef pair folded, where this layout has the key. + fn fold_pairs(&self, text: &str) -> String { + let mut out = String::with_capacity(text.len()); + let mut chars = text.chars().peekable(); + while let Some(ch) = chars.next() { + let form = chars.peek().and_then(|&next| { + let unit = |c: char| u16::try_from(u32::from(c)).ok(); + fold_lam_alef(&[unit(ch)?, unit(next)?]).filter(|&f| self.key_for_char(f).is_some()) + }); + match form { + Some(form) => { + chars.next(); + out.push(form); + } + None => out.push(ch), + } + } + out + } + /// Build the [`LayoutProfile`] used by the detector pipeline. pub fn detector_profile(&self) -> LayoutProfile { LayoutProfile::new(self.id.clone(), self.script, self.vowels.iter().copied()) diff --git a/crates/poltertype-detect/src/consts.rs b/crates/poltertype-detect/src/consts.rs index fae07d0a..c80ae8d4 100644 --- a/crates/poltertype-detect/src/consts.rs +++ b/crates/poltertype-detect/src/consts.rs @@ -41,3 +41,23 @@ pub(crate) const WEAK_PENALTY: f32 = 0.6; /// candidates that start the way the user actually typed. pub(crate) const PREFIX_BONUS: f32 = 0.08; pub(crate) const PREFIX_BONUS_CAP: usize = 4; + +/// Fewest letters a word needs before a dictionary's prefixes may move it +/// out of an abjad. Shorter stems match anything; see +/// [`crate::DictionaryDetector`] and DECISIONS.md (2026-11-04). +pub(crate) const ABJAD_ESCAPE_MIN_LETTERS: usize = 5; + +/// How much more of a word another layout's dictionary must recognise, as +/// a share of its letters, than its own does. +pub(crate) const ABJAD_ESCAPE_ADVANTAGE: f32 = 0.34; + +/// Particles Arabic writes joined to the next word, which the stems list +/// mostly lacks: a real word behind one is still the writer's own. +pub(crate) const ARABIC_CLITICS: &[&str] = &[ + "ال", "وال", "بال", "فال", "كال", "لل", "و", "ف", "ب", "ل", "ك", "س", "وب", "ول", "فب", "فل", +]; + +/// The same for Hebrew, whose stems list carries no prefixes at all. +pub(crate) const HEBREW_CLITICS: &[&str] = &[ + "ה", "ו", "ב", "כ", "ל", "מ", "ש", "וה", "וב", "ול", "שה", "שב", "של", "כש", "מה", "ומ", +]; diff --git a/crates/poltertype-detect/src/dictionary.rs b/crates/poltertype-detect/src/dictionary.rs index edea1603..11fe0145 100644 --- a/crates/poltertype-detect/src/dictionary.rs +++ b/crates/poltertype-detect/src/dictionary.rs @@ -59,6 +59,26 @@ pub struct LayoutDictionary { } impl LayoutDictionary { + /// How many leading letters of `word` begin some word of the embedded + /// list. A word this list lacks can still share a long stem with it, + /// which a whole-word lookup cannot see. + pub(crate) fn prefix_letters(&self, word: &str) -> usize { + let fst = self.embedded.as_fst(); + let mut node = fst.root(); + let mut letters = 0; + for ch in word.chars() { + let mut buf = [0u8; 4]; + for &byte in ch.encode_utf8(&mut buf).as_bytes() { + let Some(i) = node.find_input(byte) else { + return letters; + }; + node = fst.node(node.transition(i).addr); + } + letters += 1; + } + letters + } + pub fn new( embedded: FstSet<&'static [u8]>, user_overlay: HashSet, diff --git a/crates/poltertype-detect/src/dictionary_detector.rs b/crates/poltertype-detect/src/dictionary_detector.rs index 11ffc9e8..5bf4212b 100644 --- a/crates/poltertype-detect/src/dictionary_detector.rs +++ b/crates/poltertype-detect/src/dictionary_detector.rs @@ -12,6 +12,8 @@ use crate::text::{letters_only_lower, non_word_char_count, paired_segments, segm use crate::traits::Detector; use crate::types::DetectionContext; +mod abjad; + /// Looks the rendered text up in per-layout dictionaries. /// /// * `current_text` is in its layout's dictionary → `Keep` (the user @@ -339,6 +341,10 @@ impl Detector for DictionaryDetector { }; } + if !current_has_stray && let Some(switch) = self.escape_from_abjad(¤t_text, ctx) { + return switch; + } + Verdict::NoOpinion } } diff --git a/crates/poltertype-detect/src/dictionary_detector/abjad.rs b/crates/poltertype-detect/src/dictionary_detector/abjad.rs new file mode 100644 index 00000000..4b2949b4 --- /dev/null +++ b/crates/poltertype-detect/src/dictionary_detector/abjad.rs @@ -0,0 +1,81 @@ +//! Moving a word out of an Arabic or Hebrew layout on dictionary evidence. + +use poltertype_types::{DetectionVerdict, logsafe}; + +use super::DictionaryDetector; +use crate::consts::{ + ABJAD_ESCAPE_ADVANTAGE, ABJAD_ESCAPE_MIN_LETTERS, ARABIC_CLITICS, HEBREW_CLITICS, +}; +use crate::enums::{Script, Verdict}; +use crate::text::{letters_only_lower, non_word_char_count}; +use crate::types::DetectionContext; + +impl DictionaryDetector { + /// A word typed on an Arabic or Hebrew keyboard that neither dictionary + /// holds, where another layout recognises much more of it: `microsoft`, + /// which the en-US list lacks but whose stem it knows. Shape cannot + /// decide this, since a real word in an abjad reads as one consonant + /// cluster, so plausibility refuses to and the choice is made here by + /// how far each dictionary follows the word. Measured on Wikipedia text + /// in DECISIONS.md (2026-11-04). + pub(super) fn escape_from_abjad( + &self, + current: &str, + ctx: &DetectionContext<'_>, + ) -> Option { + let letters = current.chars().count(); + if letters < ABJAD_ESCAPE_MIN_LETTERS { + return None; + } + let script = Script::of(current.chars().next()?); + let clitics = match script { + Script::Arabic => ARABIC_CLITICS, + Script::Hebrew => HEBREW_CLITICS, + _ => return None, + }; + if current.chars().any(|c| Script::of(c) != script) { + return None; + } + let dicts = self.dicts.read(); + let own = dicts.get(ctx.current_layout)?; + let share = |n: usize| n as f32 / letters as f32; + let own_share = clitics + .iter() + .filter_map(|c| { + let stem = current.strip_prefix(c)?; + Some(c.chars().count() + own.prefix_letters(stem)) + }) + .chain([own.prefix_letters(current)]) + .map(share) + .fold(0.0, f32::max); + let (target, target_share) = ctx + .candidates + .iter() + // A capital is a Shift, which on these keyboards types a hamza + // form or a mark: `آخرهم` renders `Novil`. Nothing wrote it. + .filter(|(l, t)| { + l != ctx.current_layout + && non_word_char_count(t) == 0 + && !t.chars().any(char::is_uppercase) + }) + .filter_map(|(l, t)| { + let word = letters_only_lower(t); + (word.chars().count() == letters) + .then(|| Some((l, share(dicts.get(l)?.prefix_letters(&word))))) + .flatten() + }) + .max_by(|a, b| a.1.total_cmp(&b.1))?; + if target_share - own_share < ABJAD_ESCAPE_ADVANTAGE { + return None; + } + Some(Verdict::Switch(DetectionVerdict { + best_layout: target.clone(), + confidence: 0.7, + reason: format!( + "{target}'s dictionary follows {} further ({target_share:.2}) than {}'s ({own_share:.2})", + logsafe::redact_word(current), + ctx.current_layout, + ), + })) + } +} diff --git a/crates/poltertype-detect/src/enums.rs b/crates/poltertype-detect/src/enums.rs index 2d5da86f..0908a469 100644 --- a/crates/poltertype-detect/src/enums.rs +++ b/crates/poltertype-detect/src/enums.rs @@ -47,7 +47,9 @@ impl Script { 0x0370..=0x03FF => Self::Greek, 0x0530..=0x058F => Self::Armenian, 0x0590..=0x05FF => Self::Hebrew, - 0x0600..=0x06FF => Self::Arabic, + // The presentation forms count too: the lam-alef keys of an + // Arabic keyboard type U+FEFB and its siblings. + 0x0600..=0x06FF | 0x0750..=0x077F | 0xFB50..=0xFDFF | 0xFE70..=0xFEFC => Self::Arabic, _ => Self::Other, } } diff --git a/crates/poltertype-detect/src/plausibility.rs b/crates/poltertype-detect/src/plausibility.rs index b87d9281..787910f9 100644 --- a/crates/poltertype-detect/src/plausibility.rs +++ b/crates/poltertype-detect/src/plausibility.rs @@ -98,6 +98,22 @@ impl WordPlausibilityDetector { None } + /// Whether `text` is wholly letters of `layout`'s own abjad. Arabic + /// and Hebrew leave short vowels unwritten, so a correctly typed word + /// reads as one consonant cluster (`بسرعه` scores 0.0) while its + /// Latin rendering lands vowels (`fsnui` scores 0.75) and wins. + /// Shape cannot tell those apart; only the dictionary can. + fn abjad_in_own_script(&self, layout: &LayoutId, text: &str) -> bool { + let Some(prof) = self.profiles.get(layout) else { + return false; + }; + let mut letters = text.chars().filter(|c| c.is_alphabetic()).peekable(); + matches!(prof.script, Script::Arabic | Script::Hebrew) + && non_word_char_count(text) == 0 + && letters.peek().is_some() + && letters.all(|c| Script::of(c) == prof.script) + } + /// The plain word-shape score, with no compound handling. fn score(prof: &LayoutProfile, text: &str) -> f32 { let letters: Vec = text @@ -216,6 +232,19 @@ impl Detector for WordPlausibilityDetector { }; } + // Leaving an abjad is the dictionary's call: an English word typed + // on the Arabic keys is a dictionary hit there, and anything less + // is safer kept as typed than replaced on shape alone. + if self.abjad_in_own_script(ctx.current_layout, current_text) { + return Verdict::Keep { + reason: format!( + "current {} is in {}'s own abjad; left to the dictionary", + logsafe::redact_word(current_text), + ctx.current_layout + ), + }; + } + if current_fit >= self.keep_threshold { return Verdict::Keep { reason: format!( diff --git a/crates/poltertype-layout/src/linux/hyprland/tests.rs b/crates/poltertype-layout/src/linux/hyprland/tests.rs index 0567398e..bb669e3c 100644 --- a/crates/poltertype-layout/src/linux/hyprland/tests.rs +++ b/crates/poltertype-layout/src/linux/hyprland/tests.rs @@ -44,6 +44,16 @@ fn pretty_keymap_names_resolve_for_all_bundled_languages() { ("French", "fr-FR"), ("Spanish", "es-ES"), ("Spanish (Latin American)", "es-ES"), + ("Polish", "pl-PL"), + ("Czech", "cs-CZ"), + ("Greek", "el-GR"), + ("Hebrew", "he-IL"), + ("Turkish", "tr-TR"), + ("Bulgarian", "bg-BG"), + ("Italian", "it-IT"), + ("Portuguese", "pt-PT"), + ("Portuguese (Brazil)", "pt-BR"), + ("Arabic", "ar"), // "Belarusian" contains the letters "us" — it must fall // through as unknown, not resolve to en-US. ("Belarusian", "Belarusian"), diff --git a/crates/poltertype-layout/src/linux/shared.rs b/crates/poltertype-layout/src/linux/shared.rs index b92744a0..cd2b99fc 100644 --- a/crates/poltertype-layout/src/linux/shared.rs +++ b/crates/poltertype-layout/src/linux/shared.rs @@ -100,6 +100,7 @@ pub fn bcp47_to_xkb(bcp: &str) -> Option<&'static str> { "hy-AM" => "am", "ka-GE" => "ge", "he-IL" => "il", + "ar" => "ara", "ja-JP" => "jp", "ko-KR" => "kr", _ => return None, @@ -135,6 +136,16 @@ pub(crate) fn name_to_xkb_code(name: &str) -> String { // the moment the user switches to it the engine sees an unknown // current layout: empty renders, phantom re-corrections. s if s.contains("spanish") => "es".into(), + s if s.contains("polish") => "pl".into(), + s if s.contains("czech") => "cz".into(), + s if s.contains("greek") => "gr".into(), + s if s.contains("hebrew") => "il".into(), + s if s.contains("turkish") => "tr".into(), + s if s.contains("bulgarian") => "bg".into(), + s if s.contains("italian") => "it".into(), + s if s.contains("portuguese (brazil") => "br".into(), + s if s.contains("portuguese") => "pt".into(), + s if s.contains("arabic") => "ara".into(), _ => name.to_owned(), } } diff --git a/crates/poltertype-layout/src/macos.rs b/crates/poltertype-layout/src/macos.rs index 964baace..b7fd2c03 100644 --- a/crates/poltertype-layout/src/macos.rs +++ b/crates/poltertype-layout/src/macos.rs @@ -16,9 +16,11 @@ use core_foundation::base::{CFRelease, CFTypeRef, OSStatus, TCFType}; use core_foundation::dictionary::{CFDictionary, CFDictionaryRef}; use core_foundation::string::{CFString, CFStringRef}; -use crate::{LayoutError, LayoutId, LayoutSwitcher}; +use crate::{LayoutError, LayoutId, LayoutSwitcher, OsKeymap}; +mod consts; mod ids; +mod keymap; use ids::{bcp47_to_tis_id, source_id_string, source_id_to_layout_id, tis_id_to_bcp47}; // ─── Carbon FFI ────────────────────────────────────────────────────── @@ -152,6 +154,10 @@ impl LayoutSwitcher for MacosLayoutSwitcher { fn backend_name(&self) -> &'static str { "macos-tis" } + + fn describe_keymaps(&self) -> Result, LayoutError> { + run_on_main(keymap::describe_all) + } } fn current_impl() -> Result { @@ -170,8 +176,9 @@ fn current_impl() -> Result { } } -fn list_active_impl() -> Result, LayoutError> { - // Filter for keyboard sources that the user can select. +/// The keyboard sources the user can select — what `list_active` names +/// and `describe_keymaps` describes. `None` when TIS returns no list. +fn selectable_keyboards() -> Option> { let filter = unsafe { let category_key = CFString::wrap_under_get_rule(kTISPropertyInputSourceCategory); let category_val = CFString::wrap_under_get_rule(kTISCategoryKeyboardInputSource); @@ -189,17 +196,24 @@ fn list_active_impl() -> Result, LayoutError> { unsafe { let arr_ref = TISCreateInputSourceList(filter.as_concrete_TypeRef(), false); if arr_ref.is_null() { - return Ok(Vec::new()); + return None; } - let arr: CFArray = CFArray::wrap_under_create_rule(arr_ref); - let mut out = Vec::with_capacity(arr.len() as usize); - for src in arr.iter() { - if let Some(id) = source_id_to_layout_id(*src) { - out.push(id); - } + Some(CFArray::wrap_under_create_rule(arr_ref)) + } +} + +fn list_active_impl() -> Result, LayoutError> { + let Some(arr) = selectable_keyboards() else { + return Ok(Vec::new()); + }; + let mut out = Vec::with_capacity(arr.len() as usize); + for src in arr.iter() { + // Safety: `src` lives as long as `arr`. + if let Some(id) = unsafe { source_id_to_layout_id(*src) } { + out.push(id); } - Ok(out) } + Ok(out) } fn switch_to_impl(id: &LayoutId) -> Result<(), LayoutError> { @@ -236,3 +250,6 @@ fn switch_to_impl(id: &LayoutId) -> Result<(), LayoutError> { Err(LayoutError::NotActive(id.clone())) } + +#[cfg(test)] +mod tests; diff --git a/crates/poltertype-layout/src/macos/consts.rs b/crates/poltertype-layout/src/macos/consts.rs new file mode 100644 index 00000000..858930d2 --- /dev/null +++ b/crates/poltertype-layout/src/macos/consts.rs @@ -0,0 +1,55 @@ +//! Constants the macOS layout backend shares between its files. + +/// Win SC Set-1 scancode → Apple virtual keycode for the character block, +/// the same keys the bundled TOMLs list. Kept here rather than borrowed +/// from `poltertype-input`, which this crate does not depend on. +pub(super) const CHARACTER_KEYS: &[(u32, u16)] = &[ + (0x29, 0x32), // backtick + (0x02, 0x12), + (0x03, 0x13), + (0x04, 0x14), + (0x05, 0x15), + (0x06, 0x17), + (0x07, 0x16), + (0x08, 0x1A), + (0x09, 0x1C), + (0x0A, 0x19), + (0x0B, 0x1D), + (0x0C, 0x1B), + (0x0D, 0x18), + (0x10, 0x0C), // Q + (0x11, 0x0D), + (0x12, 0x0E), + (0x13, 0x0F), + (0x14, 0x11), + (0x15, 0x10), + (0x16, 0x20), + (0x17, 0x22), + (0x18, 0x1F), + (0x19, 0x23), + (0x1A, 0x21), + (0x1B, 0x1E), + (0x1E, 0x00), // A + (0x1F, 0x01), + (0x20, 0x02), + (0x21, 0x03), + (0x22, 0x05), + (0x23, 0x04), + (0x24, 0x26), + (0x25, 0x28), + (0x26, 0x25), + (0x27, 0x29), + (0x28, 0x27), + (0x2B, 0x2A), // backslash + (0x2C, 0x06), // Z + (0x2D, 0x07), + (0x2E, 0x08), + (0x2F, 0x09), + (0x30, 0x0B), + (0x31, 0x2D), + (0x32, 0x2E), + (0x33, 0x2B), + (0x34, 0x2F), + (0x35, 0x2C), + (0x56, 0x0A), // ISO key left of Z +]; diff --git a/crates/poltertype-layout/src/macos/ids.rs b/crates/poltertype-layout/src/macos/ids.rs index bf15131a..4a121dc2 100644 --- a/crates/poltertype-layout/src/macos/ids.rs +++ b/crates/poltertype-layout/src/macos/ids.rs @@ -52,6 +52,11 @@ pub(super) fn tis_id_to_bcp47(id: &str) -> Option { "com.apple.keylayout.Hebrew" => "he-IL", "com.apple.keylayout.Hebrew-PC" => "he-IL", "com.apple.keylayout.Hebrew-QWERTY" => "he-IL", + "com.apple.keylayout.Arabic" => "ar", + "com.apple.keylayout.ArabicPC" => "ar", + "com.apple.keylayout.Arabic-QWERTY" => "ar", + "com.apple.keylayout.Arabic-AZERTY" => "ar", + "com.apple.keylayout.Arabic-NorthAfrica" => "ar", "com.apple.keylayout.Turkish" => "tr-TR", "com.apple.keylayout.Turkish-Standard" => "tr-TR", "com.apple.keylayout.Turkish-QWERTY-PC" => "tr-TR", @@ -89,6 +94,7 @@ pub(super) fn bcp47_to_tis_id(id: &str) -> Option { "el-GR" => "com.apple.keylayout.Greek", "cs-CZ" => "com.apple.keylayout.Czech", "he-IL" => "com.apple.keylayout.Hebrew", + "ar" => "com.apple.keylayout.Arabic", "tr-TR" => "com.apple.keylayout.Turkish", "bg-BG" => "com.apple.keylayout.Bulgarian", "it-IT" => "com.apple.keylayout.Italian", diff --git a/crates/poltertype-layout/src/macos/keymap.rs b/crates/poltertype-layout/src/macos/keymap.rs new file mode 100644 index 00000000..0dfa1c23 --- /dev/null +++ b/crates/poltertype-layout/src/macos/keymap.rs @@ -0,0 +1,159 @@ +//! Describing keyboards: what each key of the character block produces +//! under one TIS source, read with `UCKeyTranslate` against the source's +//! `kTISPropertyUnicodeKeyLayoutData` — the macOS counterpart of the +//! Windows backend's `ToUnicodeEx` walk. +//! +//! A language is not a keyboard: Apple ships `Arabic`, `Arabic - PC` and +//! `Arabic - North Africa`, which disagree on most of the bottom row, and +//! every one of them reports the same `LayoutId`. Asking the OS is the +//! only way the DB learns which one the user actually has. + +use core_foundation::base::{CFRelease, TCFType}; +use core_foundation::data::{CFData, CFDataRef}; +use core_foundation::string::CFStringRef; +use poltertype_types::{OsKeymap, fold_lam_alef}; + +use super::consts::CHARACTER_KEYS; +use super::ids::{source_id_string, source_id_to_layout_id}; +use super::{ + TISCopyCurrentKeyboardInputSource, TISGetInputSourceProperty, TISInputSourceRef, + selectable_keyboards, +}; +use crate::LayoutError; + +#[link(name = "Carbon", kind = "framework")] +unsafe extern "C" { + static kTISPropertyUnicodeKeyLayoutData: CFStringRef; + fn UCKeyTranslate( + keyLayoutPtr: *const u8, + virtualKeyCode: u16, + keyAction: u16, + modifierKeyState: u32, + keyboardType: u32, + keyTranslateOptions: u32, + deadKeyState: *mut u32, + maxStringLength: usize, + actualStringLength: *mut usize, + unicodeString: *mut u16, + ) -> i32; + fn LMGetKbdType() -> u8; +} + +/// `kUCKeyActionDown`. +const KEY_ACTION_DOWN: u16 = 0; +/// `shiftKey >> 8`, the form `UCKeyTranslate` takes modifiers in. +const SHIFT_STATE: u32 = 0x02; +/// `kUCKeyTranslateNoDeadKeysMask`: a dead key reports its own character +/// instead of arming a compose state, which is what a key table wants. +const NO_DEAD_KEYS: u32 = 1; +/// Arabic tatweel, the kashida stretching mark. +const TATWEEL: char = '\u{0640}'; + +/// Describe one input source, or `None` when it carries no Unicode key +/// layout (input methods, palettes) and so is not a keyboard to describe. +/// +/// # Safety +/// `src` must be a live `TISInputSourceRef`. +pub(super) unsafe fn describe_source(src: TISInputSourceRef) -> Option { + let id = unsafe { source_id_to_layout_id(src) }?; + let variant = unsafe { source_id_string(src) }.unwrap_or_default(); + // Get-rule: the property is owned by the source, not by us. + let data_ref = + unsafe { TISGetInputSourceProperty(src, kTISPropertyUnicodeKeyLayoutData) } as CFDataRef; + if data_ref.is_null() { + return None; + } + let data = unsafe { CFData::wrap_under_get_rule(data_ref) }; + let layout = data.bytes().as_ptr(); + let kbd_type = u32::from(unsafe { LMGetKbdType() }); + + let mut keys = Vec::with_capacity(CHARACTER_KEYS.len()); + for &(sc, kvk) in CHARACTER_KEYS { + let Some(plain) = (unsafe { char_at(layout, kvk, 0, kbd_type) }) else { + continue; + }; + // `None` when Shift changes nothing — same convention the bundled + // TOMLs use for their optional `shift` field. + let shift = unsafe { char_at(layout, kvk, SHIFT_STATE, kbd_type) }.filter(|c| *c != plain); + keys.push((sc, plain, shift)); + } + tracing::debug!(%id, %variant, keys = keys.len(), "described keyboard"); + Some(OsKeymap { id, variant, keys }) +} + +/// The single printable character a key produces, or `None` for nothing, +/// a control character, or any other multi-character output — a key +/// table entry is one `char`. Arabic's lam-alef keys type two code points +/// and are folded to their presentation form, as `ar.toml` does. +/// +/// Tatweel (U+0640) is `None` too, as it is in `ar.toml`: it is +/// alphabetic but stretches a word rather than spelling one, and Apple's +/// Arabic keyboards put it on Shift+`-`, where it would make `_` read as +/// a letter and switch off the snake_case heuristic. +/// +/// # Safety +/// `layout` must point at a live `UCKeyboardLayout`. +unsafe fn char_at(layout: *const u8, kvk: u16, modifiers: u32, kbd_type: u32) -> Option { + let mut dead_state = 0u32; + let mut len = 0usize; + let mut buf = [0u16; 4]; + let status = unsafe { + UCKeyTranslate( + layout, + kvk, + KEY_ACTION_DOWN, + modifiers, + kbd_type, + NO_DEAD_KEYS, + &mut dead_state, + buf.len(), + &mut len, + buf.as_mut_ptr(), + ) + }; + if status != 0 || len == 0 { + return None; + } + if let Some(ligature) = fold_lam_alef(&buf[..len]) { + return Some(ligature); + } + let mut chars = char::decode_utf16(buf[..len].iter().copied()); + let c = chars.next()?.ok()?; + if chars.next().is_some() || c.is_control() || c == TATWEEL { + return None; + } + Some(c) +} + +/// Every selectable keyboard, described; the one in effect comes first. +pub(super) fn describe_all() -> Result, LayoutError> { + let Some(arr) = selectable_keyboards() else { + return Ok(Vec::new()); + }; + // Safety: a copied source is retained and released right after its + // id is read; the sources in `arr` live as long as `arr`. + let current = unsafe { + let src = TISCopyCurrentKeyboardInputSource(); + if src.is_null() { + None + } else { + let id = source_id_string(src); + CFRelease(src); + id + } + }; + let mut out: Vec = arr + .iter() + .filter_map(|src| unsafe { describe_source(*src) }) + .collect(); + + // The keyboard in effect right now goes first: two keyboards of one + // language collapse to one `LayoutId`, and the DB keeps the first. + if let Some(pos) = out + .iter() + .position(|k| Some(&k.variant) == current.as_ref()) + { + out.swap(0, pos); + } + Ok(out) +} diff --git a/crates/poltertype-layout/src/macos/tests.rs b/crates/poltertype-layout/src/macos/tests.rs new file mode 100644 index 00000000..7496e931 --- /dev/null +++ b/crates/poltertype-layout/src/macos/tests.rs @@ -0,0 +1,42 @@ +//! The TIS calls themselves must run on the main dispatch queue, which a +//! test thread cannot reach, so these pin the key table the describer +//! walks rather than the OS answer. + +#![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)] + +use super::consts::CHARACTER_KEYS; + +/// Every scancode a bundled layout lists must be one the describer asks +/// about — a key it never asks about is a key an OS keymap silently +/// drops when it replaces the bundled table. +#[test] +fn character_keys_cover_every_bundled_scancode() { + let path = concat!( + env!("CARGO_MANIFEST_DIR"), + "/../../data/layout-mappings/en_us.toml" + ); + let toml = std::fs::read_to_string(path).expect("read en_us.toml"); + for line in toml.lines().filter(|l| l.starts_with("0x")) { + let hex = line.split_whitespace().next().unwrap_or_default(); + let sc = u32::from_str_radix(hex.trim_start_matches("0x"), 16).expect("hex scancode"); + assert!( + CHARACTER_KEYS.iter().any(|&(k, _)| k == sc), + "0x{sc:02X} is bundled but never described" + ); + } +} + +/// One row per scancode and per Apple keycode — a duplicate would +/// describe one physical key twice under two names. +#[test] +fn character_keys_are_one_to_one() { + let mut scancodes = std::collections::HashSet::new(); + let mut keycodes = std::collections::HashSet::new(); + for &(sc, kvk) in CHARACTER_KEYS { + assert!(scancodes.insert(sc), "SC-1 0x{sc:02X} listed twice"); + assert!( + keycodes.insert(kvk), + "Apple keycode 0x{kvk:02X} listed twice" + ); + } +} diff --git a/crates/poltertype-layout/src/windows.rs b/crates/poltertype-layout/src/windows.rs index 675ff6d0..bc3587b1 100644 --- a/crates/poltertype-layout/src/windows.rs +++ b/crates/poltertype-layout/src/windows.rs @@ -111,11 +111,20 @@ impl LayoutSwitcher for WindowsLayoutSwitcher { fn switch_to(&self, id: &LayoutId) -> Result<(), LayoutError> { let active = self.list_active()?; - let active_map: HashMap = self - .raw_active()? - .into_iter() - .map(|h| (hkl_to_layout_id(h), h)) - .collect(); + // First per id, the one in effect ahead of the rest: the same + // keyboard `describe_keymaps` gave the DB its table from. Two + // keyboards of one language must not switch to the other's keys. + let mut hkls = self.raw_active()?; + // Safety: pure Win32 calls with no aliasing concerns. + let current = + unsafe { GetKeyboardLayout(GetWindowThreadProcessId(GetForegroundWindow(), None)) }; + if let Some(pos) = hkls.iter().position(|h| h.0 == current.0) { + hkls.swap(0, pos); + } + let mut active_map: HashMap = HashMap::new(); + for h in hkls { + active_map.entry(hkl_to_layout_id(h)).or_insert(h); + } let Some(&hkl) = active_map.get(id) else { warn!(?id, ?active, "requested layout is not active"); @@ -194,12 +203,24 @@ fn hkl_to_layout_id(hkl: HKL) -> LayoutId { let n = (written as usize).saturating_sub(1); // strip trailing NUL let s = String::from_utf16_lossy(&buf[..n]); if !s.is_empty() { - return LayoutId::new(s); + return LayoutId::new(language_of_locale(s)); } } LayoutId::new(format!("hkl:{raw:08x}")) } +/// The id a Windows locale name goes by. Arabic is bundled as one +/// language, `ar`, and Windows names a country for every keyboard: +/// `ar-SA`, `ar-EG`, `ar-001`. Which keyboard it is comes from +/// [`describe_hkl`] anyway. +fn language_of_locale(locale: String) -> String { + if locale.split('-').next() == Some("ar") { + "ar".to_owned() + } else { + locale + } +} + /// Describe one keyboard: what each key of the character block /// produces, unshifted and shifted. /// @@ -270,6 +291,11 @@ fn char_at(vk: u32, sc: u32, hkl: HKL, shift: bool) -> Option { } else { (n as usize).min(buf.len()) }; + // An Arabic keyboard answers its lam-alef keys with two code points; + // the first alone would turn `B` into `ل`. + if let Some(ligature) = poltertype_types::fold_lam_alef(&buf[..len]) { + return Some(ligature); + } String::from_utf16_lossy(&buf[..len]) .chars() .next() diff --git a/crates/poltertype-layout/src/windows/tests.rs b/crates/poltertype-layout/src/windows/tests.rs index a1ab9987..befc8df7 100644 --- a/crates/poltertype-layout/src/windows/tests.rs +++ b/crates/poltertype-layout/src/windows/tests.rs @@ -167,3 +167,15 @@ fn the_three_bulgarian_keyboards_describe_differently() { is not variant-aware after all" ); } + +/// A bundled `ar` is only found if every country's Arabic keyboard goes +/// by it; other languages keep their region. +#[test] +fn every_arabic_locale_is_the_one_arabic_layout() { + for locale in ["ar-SA", "ar-EG", "ar-001", "ar"] { + assert_eq!(language_of_locale(locale.to_owned()), "ar", "{locale}"); + } + for locale in ["en-US", "uk-UA", "arn-CL"] { + assert_eq!(language_of_locale(locale.to_owned()), locale); + } +} diff --git a/crates/poltertype-types/src/lam_alef.rs b/crates/poltertype-types/src/lam_alef.rs new file mode 100644 index 00000000..e3b66a92 --- /dev/null +++ b/crates/poltertype-types/src/lam_alef.rs @@ -0,0 +1,33 @@ +//! The one case of a key that types two code points. +//! +//! A key table entry is one `char`, and Arabic's lam-alef keys break that: +//! macOS and Windows answer `B` with `لا`, U+0644 U+0627. On Linux xkb's +//! `ara` emits the presentation form U+FEFB and libX11's Compose table, +//! which GTK honours, turns that back into the same pair. So the table +//! carries the form, and the screen holds two code points. + +const LIGATURES: [(u16, char); 4] = [ + (0x0627, '\u{FEFB}'), + (0x0623, '\u{FEF7}'), + (0x0625, '\u{FEF9}'), + (0x0622, '\u{FEF5}'), +]; +const LAM: u16 = 0x0644; + +/// The presentation form of a lam-alef pair, for a backend reading what a +/// key produces. `None` for anything else. +pub fn fold_lam_alef(utf16: &[u16]) -> Option { + match utf16 { + [LAM, alef] => LIGATURES + .iter() + .find(|(a, _)| a == alef) + .map(|&(_, form)| form), + _ => None, + } +} + +/// The two code points a presentation form stands for. +pub fn lam_alef_pair(form: char) -> Option<[char; 2]> { + let &(alef, _) = LIGATURES.iter().find(|&&(_, f)| f == form)?; + Some(['\u{0644}', char::from_u32(u32::from(alef))?]) +} diff --git a/crates/poltertype-types/src/lib.rs b/crates/poltertype-types/src/lib.rs index 95299fc2..d2d45905 100644 --- a/crates/poltertype-types/src/lib.rs +++ b/crates/poltertype-types/src/lib.rs @@ -8,11 +8,13 @@ pub mod ai_plugin; mod consts; mod enums; +mod lam_alef; pub mod logsafe; mod types; pub use consts::*; pub use enums::*; +pub use lam_alef::{fold_lam_alef, lam_alef_pair}; pub use types::*; #[cfg(test)] diff --git a/crates/poltertype-types/src/tests.rs b/crates/poltertype-types/src/tests.rs index c915f89f..2d34f857 100644 --- a/crates/poltertype-types/src/tests.rs +++ b/crates/poltertype-types/src/tests.rs @@ -71,3 +71,19 @@ fn redaction_hides_the_word_and_keeps_its_length() { fn opted_in_debug_shows_the_word_in_backticks() { assert_eq!(logsafe::render("mañana", true), "`mañana`"); } + +#[test] +fn lam_alef_folds_and_unfolds_each_form() { + for (alef, form) in [ + ('\u{0627}', '\u{FEFB}'), + ('\u{0623}', '\u{FEF7}'), + ('\u{0625}', '\u{FEF9}'), + ('\u{0622}', '\u{FEF5}'), + ] { + assert_eq!(crate::fold_lam_alef(&[0x0644, alef as u16]), Some(form)); + assert_eq!(crate::lam_alef_pair(form), Some(['\u{0644}', alef])); + } + assert_eq!(crate::fold_lam_alef(&[0x0644]), None); + assert_eq!(crate::fold_lam_alef(&[0x0627, 0x0644]), None); + assert_eq!(crate::lam_alef_pair('a'), None); +} diff --git a/data/layout-mappings/ar.toml b/data/layout-mappings/ar.toml new file mode 100644 index 00000000..2e6eed8e --- /dev/null +++ b/data/layout-mappings/ar.toml @@ -0,0 +1,98 @@ +# Layout mapping: العربية (Arabic, standard "Arabic (101)"). +# +# Keyed by Win SC Set-1 scancode (hex). This table only speaks for Linux — +# macOS and Windows read the installed keyboard from the OS and replace it +# — so it follows xkeyboard-config's `ara`, checked key by key against +# libxkbcommon: ASCII digits on the number row, `%` on Shift+5. Arabic is +# unicase; the shifted level carries harakat, hamza forms and punctuation. +# +# The lam-alef keys produce a two-code-point ligature (`لا` on 0x30 `B`, +# `لإ` / `لأ` on shifted 0x14 / 0x22, `لآ` on shifted `B`), and a key +# here is one character. They carry the single-code-point presentation +# forms instead (U+FEFB, U+FEF9, U+FEF7, U+FEF5), which render the same. +# On screen they stay two code points on every OS, xkb's Compose table +# included, and the engine erases them as two. +# Leaving them out is worse: an unmapped key is dropped from a +# translation, so `gdb` would read as `لي`, a real word, and `B` would +# vanish from every corrected word. The wordlist carries each `لا` word +# in this spelling too, so `hgsbl` still finds `السلام`. +# +# And one character is kept off the table on purpose: tatweel `ـ` +# (U+0640) is Unicode-alphabetic but a stretching mark, not a letter. +# Listed on Shift+`-` it would make `_` read as a letter in some layout +# and switch off the snake_case heuristic for everyone, so 0x0C keeps +# Arabic 101's `_` and 0x24 carries no shifted level. +# +# Apple's own `Arabic` / `Arabic - North Africa` layouts differ from this +# one on several letter keys (B, N, M, comma, period, slash and the +# brackets); all Mac variants map to this id, and the macOS backend +# reads the keyboard actually installed from the OS, which outranks +# this table. +# +# The plausibility detector counts Arabic's written long vowels (alef, +# waw, yeh, alef maksura, and the lam-alef keys) — see `derive_vowels` — +# and the wordlist makes ar ↔ en-US sharp. No other bundled layout uses +# the Arabic script. + +id = "ar" +name = "العربية" +script = "Arabic" + +[keys] + +# Backtick +0x29 = { plain = "ذ", shift = "ّ" } + +# Number row +0x02 = { plain = "1", shift = "!" } +0x03 = { plain = "2", shift = "@" } +0x04 = { plain = "3", shift = "#" } +0x05 = { plain = "4", shift = "$" } +0x06 = { plain = "5", shift = "%" } +0x07 = { plain = "6", shift = "^" } +0x08 = { plain = "7", shift = "&" } +0x09 = { plain = "8", shift = "*" } +0x0A = { plain = "9", shift = ")" } +0x0B = { plain = "0", shift = "(" } +0x0C = { plain = "-", shift = "_" } +0x0D = { plain = "=", shift = "+" } + +# Top row +0x10 = { plain = "ض", shift = "َ" } +0x11 = { plain = "ص", shift = "ً" } +0x12 = { plain = "ث", shift = "ُ" } +0x13 = { plain = "ق", shift = "ٌ" } +0x14 = { plain = "ف", shift = "ﻹ" } +0x15 = { plain = "غ", shift = "إ" } +0x16 = { plain = "ع", shift = "`" } +0x17 = { plain = "ه", shift = "÷" } +0x18 = { plain = "خ", shift = "×" } +0x19 = { plain = "ح", shift = "؛" } +0x1A = { plain = "ج", shift = "<" } +0x1B = { plain = "د", shift = ">" } + +# Home row +0x1E = { plain = "ش", shift = "ِ" } +0x1F = { plain = "س", shift = "ٍ" } +0x20 = { plain = "ي", shift = "]" } +0x21 = { plain = "ب", shift = "[" } +0x22 = { plain = "ل", shift = "ﻷ" } +0x23 = { plain = "ا", shift = "أ" } +0x24 = { plain = "ت" } +0x25 = { plain = "ن", shift = "،" } +0x26 = { plain = "م", shift = "/" } +0x27 = { plain = "ك", shift = ":" } +0x28 = { plain = "ط", shift = "\"" } + +# Bottom row +0x2B = { plain = "\\", shift = "|" } +0x2C = { plain = "ئ", shift = "~" } +0x2D = { plain = "ء", shift = "ْ" } +0x2E = { plain = "ؤ", shift = "}" } +0x2F = { plain = "ر", shift = "{" } +0x30 = { plain = "ﻻ", shift = "ﻵ" } +0x31 = { plain = "ى", shift = "آ" } +0x32 = { plain = "ة", shift = "'" } +0x33 = { plain = "و", shift = "," } +0x34 = { plain = "ز", shift = "." } +0x35 = { plain = "ظ", shift = "؟" } diff --git a/data/wordlists/CREDITS.md b/data/wordlists/CREDITS.md index 33d81f40..8667c568 100644 --- a/data/wordlists/CREDITS.md +++ b/data/wordlists/CREDITS.md @@ -71,6 +71,7 @@ was wrong about at least Russian, which is BSD. | `it_it` | `it_IT/it_IT.dic` | GPL-3 | 3.3 M | | `pt_pt` | `pt_PT/pt_PT.dic` | GPL-2-or-later | 1.3 M | | `pt_br` | `pt_BR/pt_BR.dic` | LGPL-3 / MPL | 8.1 M | +| `ar` | `ar/ar.dic` | GPL-2.0+ / LGPL-2.1+ / MPL-1.1+ (Ayaspell; pick one) | 2.5 M | ### Hebrew is the row to look at twice @@ -88,6 +89,20 @@ plausibility detector separates he-IL from every other layout on script alone and the dictionary is a refinement, not the load-bearing signal. +### Arabic ships stems too + +Ayaspell's `ar.aff` compresses its flags into a 333-entry `AF` alias +table under `FLAG long`, and its rules attach the article and the +clitic particles (ال و ف ب ل ك) as prefixes — the same combinatorial +shape as Hebrew. `ar` is therefore processed with +`ExpandMode::StemsOnly`, and the article forms come back through the +source's `Extras` instead of the affix rules: every stem is also +emitted behind `ال`, `وال`, `بال`, `فال`, `كال` and `لل`, then every +form holding `لا` (or `لأ`, `لإ`, `لآ`) also in the one-code-point +spelling the `B` key types. That gives 2.5 M forms from 466 k `.dic` +lines. A word with a bare one-letter clitic (`وكتاب`, `بسم`) still +misses the dictionary and is scored on plausibility. + ### Hebrew also ships stems, not surface forms `he_IL.aff` is not an inflection table — it is 3335 **prefix** rules diff --git a/data/wordlists/ar-stop.txt b/data/wordlists/ar-stop.txt new file mode 100644 index 00000000..f88eafe5 --- /dev/null +++ b/data/wordlists/ar-stop.txt @@ -0,0 +1,77 @@ +# Hand-curated 1- and 2-letter Arabic stop words (compile-time +# baseline). +# +# `DictionaryDetector` consults this list **instead of** the embedded +# FST for buffers ≤2 letters long, because the FST ships too many +# spurious short "words" that would block real corrections. +# +# Runtime extension (no rebuild needed): drop matching files into +# `/poltertype/wordlists/`: +# * `ar-stop.txt` → merged into THIS list at startup / reload +# * `ar-extras.txt` → merged into the user-overlay HashSet +# * `ar.txt` → same as -extras, just a different filename +# All three accept the same format and reload via "Reload Settings". +# +# Format: one word per line; blank lines and `#` comments ignored. +# +# Every entry was checked against `en_us-stop.txt` through the Arabic +# (101) key positions: none of them is what an English stop word types +# as on this layout, so none can veto an English → Arabic correction of +# a real English word. (`فخ` = "to" and `هب` = "if" are left out for +# exactly that reason.) + +# 1-letter — deliberately empty. Arabic's one-letter forms +# (و ف ب ل ك) are mostly prefixes written attached to the next word, +# and listing them would veto legitimate corrections. + +# 2-letter — particles, pronouns, prepositions and a few common nouns. +# Hamza spellings are listed both ways (أن / ان) because informal +# typing often drops the hamza. +في +من +عن +ما +لا +لم +لن +إن +أن +ان +إذ +اذ +هو +هي +هم +هن +مع +يا +كم +قد +ثم +أو +او +بل +لو +كل +أي +اي +به +بك +بي +له +لك +لي +أب +اب +أخ +اخ +أم +ام +يد +دم +فم +عم +حب +سن +جو +ذا diff --git a/data/wordlists/ar.txt.gz b/data/wordlists/ar.txt.gz new file mode 100644 index 00000000..185a81f1 Binary files /dev/null and b/data/wordlists/ar.txt.gz differ diff --git a/docs/ADDING_A_LANGUAGE.md b/docs/ADDING_A_LANGUAGE.md index a0837a28..866a3788 100644 --- a/docs/ADDING_A_LANGUAGE.md +++ b/docs/ADDING_A_LANGUAGE.md @@ -42,7 +42,7 @@ script = "Latin" # Latin / Cyrillic / Greek / Armenian / Hebrew / Arabi ### Which scancodes matter Cover at least the alphanumeric block plus the punctuation that -serves as a word boundary in your language. The fifteen bundled files +serves as a word boundary in your language. The sixteen bundled files in `data/layout-mappings/` are good copy-paste templates — they all list the same 46 scancodes (number row, three letter rows, and the key left of `Z`), plus `0x56` where a layout puts a letter on the diff --git a/docs/DECISIONS.md b/docs/DECISIONS.md index 0a14a1e1..46a7e13a 100644 --- a/docs/DECISIONS.md +++ b/docs/DECISIONS.md @@ -5074,3 +5074,52 @@ per-platform limits, and Off-by-default direct arrows already keep the arrows the user's unless they opt in. Per-suggestion custom bindings (#78's third option) would be a key-binding language for nine rows; two fixed sets cover the conflict the issue reports. + +## 2026-11-04 — A key table entry stays one `char`; lam-alef is a fact the engine knows + +Arabic's `B` key types `لا`, two code points, and a key table holds one +`char`. PR #83 had the table carry the presentation form `ﻻ`. That keeps +translation and dictionary lookups simple, but the erase counts assumed one +code point per key, which is what `Keys, not rendered chars` had always +meant: in the guest, `table` typed on Arabic came back `فtable`. + +The first fix had backends report which keys were pairs, on the theory that +xkb really types U+FEFB and only macOS and Windows produce two. Measuring +killed the theory: libX11's Compose table maps `` to `لا`, so GTK and +xterm hold two code points as well. A per-keyboard field then described a +difference that does not exist, and it came out again. + +**Decision:** the four forms mean a pair everywhere, in +`poltertype_types::lam_alef`, and `LayoutMapping::extra_backspaces` turns +that into an erase count. Widening the table to `String` would have been the +general answer for a case that has exactly one instance, across every +reader of `keys`. + +**Cost:** an application that shows U+FEFB uncomposed loses one character +more than it should. None measured here does. Selection text is ambiguous +the other way: `لا` reads back as `B`, not `G` + `H`, because `B` is by far +the commoner source. + +## 2026-11-04 — Leaving an abjad: how far each dictionary follows the word + +An abjad leaves short vowels unwritten, so a real word in Arabic or Hebrew +reads as one consonant cluster and its Latin rendering usually scores better +on shape. PR #83 stopped plausibility from deciding here at all. Measured on +Wikipedia, tokens of three letters or more that this would otherwise replace: + +| Rule | Arabic words damaged (of 90,086) | Hebrew words damaged (of 74,071) | English out of dictionary corrected | +|---|---|---|---| +| shape alone, as before #83 | 816 | 14,319 (19 %) | ~80–90 % | +| shape, with stems after a clitic kept | 554 | 1,164 | ~80–90 % | +| the veto alone, as in #83 | 0 | 0 | 0 | +| **prefix share ≥ 0.34, 5+ letters** | **0** | **0** | **~40 %** | + +**Decision:** keep the veto in plausibility, and let `DictionaryDetector` move +a word only when another layout's FST follows at least 0.34 more of its letters +than its own does, counting the own side after stripping a clitic. It is a +question only a dictionary can answer, since a real word usually shares a stem +with the list even when it is missing from it. + +**Cost:** most unknown English stays as typed on those keyboards; a word put +right by hand costs a keypress and a word destroyed costs trust. Common +Arabic letter runs defeat it (`linux`, `debian`). diff --git a/docs/KNOWN-GAPS.md b/docs/KNOWN-GAPS.md index 3ee7fcaa..912ba0ab 100644 --- a/docs/KNOWN-GAPS.md +++ b/docs/KNOWN-GAPS.md @@ -1,4 +1,4 @@ -# Known gaps (as of v0.38.2) +# Known gaps (as of v0.39.0) Things a reader of the docs might reasonably assume work, but don't. Check here before promising any of them (especially on the website). @@ -13,6 +13,22 @@ sentence above exists to prevent. ## What each release pass actually checked +**What the 0.39.0 pass actually checked (2026-11-04).** Arabic in a +real session, in the desktop-matrix guest on openbox X11, against xterm +and a GTK text field, read back byte for byte: both directions of a +correction, a word typed with the lam-alef key, a suggestion accepted on +one, the manual hotkey undoing and redoing one, and a selection converted +with the clipboard restored. The tooltip and the Languages pane were +looked at. sway recognises `Arabic` as `ar`, but that rig scrambles +keystrokes with or without Arabic, so nothing else was read from it. +Windows and macOS were compiled, not run: `describe_keymaps` on macOS, +the `ar-*` mapping and `switch_to` change on Windows. The chime on a +converted selection reached the guest's sound card (the PCM ran around the +conversion with the chime on, 69 of 93 polls, and never with it off, 0 of +77). Out-of-dictionary words were measured on about 90,000 Arabic, 74,000 +Hebrew and 3,600 out-of-dictionary English words from Wikipedia through the +real detectors. Every other bullet was re-read and not re-measured. + **What the 0.38.2 pass actually checked (2026-10-19).** The accept chord held past the digit, in the desktop-matrix guest with a debug build of the release code: `Ctrl+Meta` held 1.5 s typed 0.38.1's @@ -1182,7 +1198,7 @@ move too fast to be true. ## Languages and keyboard data -- **Nine of the fifteen bundled layouts have never been typed on.** +- **Nine of the sixteen bundled layouts have never been typed on.** 0.9.0 added pl, cs, el, he, tr, bg, it, pt-PT and pt-BR. The mappings were generated from `xkeyboard-config` rather than transcribed by hand — which is why they are trustworthy at all, and @@ -1221,6 +1237,34 @@ move too fast to be true. A report that a correction produces the wrong letter under one of these is a data bug in one TOML, not an engine bug — fix the row. +- **Arabic (new in this release) has been typed on Linux only.** On + X11 (openbox, xterm and a GTK text field) both directions, a suggestion, + the manual hotkey and a selection were checked byte for byte; on sway the + layout is recognised and corrections go through, though that rig + scrambles keystrokes with or without Arabic, so its text proves little. + macOS was read on the contributor's Mac (`Arabic - North Africa`, 47 keys + adopted). Windows has not run at all: its `ar-*` mapping and lam-alef fold + are unit-tested and compile, nothing more. + The lam-alef keys carry the limits. One key puts two code points on + screen, which a correction erases with two Backspaces: measured under + GTK and xterm, where libX11's Compose table makes the pair, and assumed + on macOS and Windows. An application that shows xkb's U+FEFB uncomposed + would lose one character more. Converting a selection reads any + `لا` as that one key, so `gh` typed on `G` and `H` comes back as `b`. + Words with a bare one-letter clitic (`وكتاب`) miss the dictionary. On an + Arabic or Hebrew keyboard about 60 % of out-of-dictionary English still + stays as typed (`debian`, `ubuntu`, `linux` on Arabic), because keeping a + real word matters more. A real word whose rendering is itself in the + English list (`שביקר` → `acher`, about 4 in 10,000 Arabic and 22 in 10,000 + Hebrew words on Wikipedia) is still switched: that is the dictionary sweep + every layout has, not this rule. + +- **On macOS the OS answers too, since this release.** `describe_keymaps` + reads each keyboard with `UCKeyTranslate` and replaces the bundled table, + for **every** macOS user rather than only Arabic ones. It compiles and + passes in CI, and was seen working on one Mac; a wrong table there is + fixed by a TOML in `/poltertype/layouts/`, which outranks it. + - **On Windows a language is not a keyboard — handled since 0.14.3, but not all the way down.** `hkl_to_layout_id` still keeps only the LCID, so all three of Windows' genuinely different Bulgarian diff --git a/docs/PLAN.md b/docs/PLAN.md index ef7c7602..d1bcb05a 100644 --- a/docs/PLAN.md +++ b/docs/PLAN.md @@ -1,7 +1,7 @@ # PolterType — Project Plan > A living roadmap. Updated as implementation proceeds. -> Created: 2026-05-02. Last updated: 2026-10-19 (v0.38.2). +> Created: 2026-05-02. Last updated: 2026-11-04 (v0.39.0). > **How to read this document.** This is a **plan**, not a description > of the implementation: wherever the code has diverged from the @@ -888,7 +888,7 @@ Levels: ## 10. Roadmap -> **Status as of v0.38.2 (2026-10-19).** Phases 0–8 are, in their core +> **Status as of v0.39.0 (2026-11-04).** Phases 0–8 are, in their core > parts, complete and shipped; `CHANGELOG.md` is the release-by-release > record and does not need repeating here. Items that are **not** done > are deliberately left as diff --git a/installers/wix/main.wxs b/installers/wix/main.wxs index 32f219b5..a8f0bd34 100644 --- a/installers/wix/main.wxs +++ b/installers/wix/main.wxs @@ -180,6 +180,9 @@ + @@ -269,6 +272,9 @@ + @@ -315,6 +321,9 @@ + @@ -362,6 +371,9 @@ + diff --git a/xtask/src/consts.rs b/xtask/src/consts.rs index 30ac1155..f4837a65 100644 --- a/xtask/src/consts.rs +++ b/xtask/src/consts.rs @@ -1,7 +1,7 @@ //! Upstream dictionary source URLs. use crate::enums::ExpandMode; -use crate::types::HunspellSource; +use crate::types::{Extras, HunspellSource}; pub(crate) const EN_URL: &str = "https://raw.githubusercontent.com/dwyl/english-words/master/words_alpha.txt"; @@ -9,6 +9,13 @@ pub(crate) const EN_URL: &str = pub(crate) const UK_README_URL: &str = "https://raw.githubusercontent.com/LibreOffice/dictionaries/master/uk_UA/README_uk_UA.txt"; +/// The [`Extras`] of a dictionary that needs none. +pub(crate) const NO_EXTRAS: Extras = Extras { + prefixes: &[], + respell: &[], + drop: &[], +}; + /// The Hunspell-derived bundled languages, one [`HunspellSource`] each. /// /// Without affix expansion ~70 % of common verb and declension forms @@ -24,6 +31,7 @@ pub(crate) const HUNSPELL_SOURCES: &[HunspellSource] = &[ aff: "https://raw.githubusercontent.com/LibreOffice/dictionaries/master/uk_UA/uk_UA.aff", output: "uk_ua.txt.gz", expand: ExpandMode::Full, + extras: NO_EXTRAS, }, HunspellSource { base: "ru_RU", @@ -31,6 +39,7 @@ pub(crate) const HUNSPELL_SOURCES: &[HunspellSource] = &[ aff: "https://raw.githubusercontent.com/LibreOffice/dictionaries/master/ru_RU/ru_RU.aff", output: "ru_ru.txt.gz", expand: ExpandMode::Full, + extras: NO_EXTRAS, }, HunspellSource { base: "de_DE_frami", @@ -38,6 +47,7 @@ pub(crate) const HUNSPELL_SOURCES: &[HunspellSource] = &[ aff: "https://raw.githubusercontent.com/LibreOffice/dictionaries/master/de/de_DE_frami.aff", output: "de_de.txt.gz", expand: ExpandMode::Full, + extras: NO_EXTRAS, }, HunspellSource { base: "es_ES", @@ -45,6 +55,7 @@ pub(crate) const HUNSPELL_SOURCES: &[HunspellSource] = &[ aff: "https://raw.githubusercontent.com/LibreOffice/dictionaries/master/es/es_ES.aff", output: "es_es.txt.gz", expand: ExpandMode::Full, + extras: NO_EXTRAS, }, // French moved down a level upstream (`fr_FR/` → `fr_FR/ // dictionaries/`) and the old URLs 404'd for months, because the @@ -57,6 +68,7 @@ pub(crate) const HUNSPELL_SOURCES: &[HunspellSource] = &[ aff: "https://raw.githubusercontent.com/LibreOffice/dictionaries/master/fr_FR/dictionaries/fr.aff", output: "fr_fr.txt.gz", expand: ExpandMode::Full, + extras: NO_EXTRAS, }, HunspellSource { base: "pl_PL", @@ -64,6 +76,7 @@ pub(crate) const HUNSPELL_SOURCES: &[HunspellSource] = &[ aff: "https://raw.githubusercontent.com/LibreOffice/dictionaries/master/pl_PL/pl_PL.aff", output: "pl_pl.txt.gz", expand: ExpandMode::Full, + extras: NO_EXTRAS, }, HunspellSource { base: "cs_CZ", @@ -71,6 +84,7 @@ pub(crate) const HUNSPELL_SOURCES: &[HunspellSource] = &[ aff: "https://raw.githubusercontent.com/LibreOffice/dictionaries/master/cs_CZ/cs_CZ.aff", output: "cs_cz.txt.gz", expand: ExpandMode::Full, + extras: NO_EXTRAS, }, HunspellSource { base: "el_GR", @@ -78,6 +92,7 @@ pub(crate) const HUNSPELL_SOURCES: &[HunspellSource] = &[ aff: "https://raw.githubusercontent.com/LibreOffice/dictionaries/master/el_GR/el_GR.aff", output: "el_gr.txt.gz", expand: ExpandMode::Full, + extras: NO_EXTRAS, }, HunspellSource { base: "he_IL", @@ -85,6 +100,29 @@ pub(crate) const HUNSPELL_SOURCES: &[HunspellSource] = &[ aff: "https://raw.githubusercontent.com/LibreOffice/dictionaries/master/he_IL/he_IL.aff", output: "he_il.txt.gz", expand: ExpandMode::StemsOnly, + extras: NO_EXTRAS, + }, + HunspellSource { + base: "ar", + dic: "https://raw.githubusercontent.com/LibreOffice/dictionaries/master/ar/ar.dic", + aff: "https://raw.githubusercontent.com/LibreOffice/dictionaries/master/ar/ar.aff", + output: "ar.txt.gz", + expand: ExpandMode::StemsOnly, + extras: Extras { + // The definite article and the clitics that fuse with it. + // Stems carry none of them, and `ال` starts a large share + // of running Arabic text. Safe against English: every one + // ends in `hg` / `gg` on the en-US keys, which no English + // word starts with. + prefixes: &["ال", "وال", "بال", "فال", "كال", "لل"], + // The lam-alef keys type one presentation-form code point + // (see `ar.toml`), so `hgsbl` renders as `السﻻم`. + respell: &[("لا", "ﻻ"), ("لأ", "ﻷ"), ("لإ", "ﻹ"), ("لآ", "ﻵ")], + // `حصي` is what `pwd` types as on Arabic 101, and the + // standard spelling of "pebbles" is `حصى` — kept out so the + // shell entry in `en_us-extras.txt` does not veto it. + drop: &["حصي"], + }, }, HunspellSource { base: "tr_TR", @@ -92,6 +130,7 @@ pub(crate) const HUNSPELL_SOURCES: &[HunspellSource] = &[ aff: "https://raw.githubusercontent.com/LibreOffice/dictionaries/master/tr_TR/tr_TR.aff", output: "tr_tr.txt.gz", expand: ExpandMode::Full, + extras: NO_EXTRAS, }, HunspellSource { base: "bg_BG", @@ -99,6 +138,7 @@ pub(crate) const HUNSPELL_SOURCES: &[HunspellSource] = &[ aff: "https://raw.githubusercontent.com/LibreOffice/dictionaries/master/bg_BG/bg_BG.aff", output: "bg_bg.txt.gz", expand: ExpandMode::Full, + extras: NO_EXTRAS, }, HunspellSource { base: "it_IT", @@ -106,6 +146,7 @@ pub(crate) const HUNSPELL_SOURCES: &[HunspellSource] = &[ aff: "https://raw.githubusercontent.com/LibreOffice/dictionaries/master/it_IT/it_IT.aff", output: "it_it.txt.gz", expand: ExpandMode::Full, + extras: NO_EXTRAS, }, HunspellSource { base: "pt_PT", @@ -113,6 +154,7 @@ pub(crate) const HUNSPELL_SOURCES: &[HunspellSource] = &[ aff: "https://raw.githubusercontent.com/LibreOffice/dictionaries/master/pt_PT/pt_PT.aff", output: "pt_pt.txt.gz", expand: ExpandMode::Full, + extras: NO_EXTRAS, }, HunspellSource { base: "pt_BR", @@ -120,6 +162,7 @@ pub(crate) const HUNSPELL_SOURCES: &[HunspellSource] = &[ aff: "https://raw.githubusercontent.com/LibreOffice/dictionaries/master/pt_BR/pt_BR.aff", output: "pt_br.txt.gz", expand: ExpandMode::Full, + extras: NO_EXTRAS, }, ]; diff --git a/xtask/src/types.rs b/xtask/src/types.rs index 3c420854..3a869d27 100644 --- a/xtask/src/types.rs +++ b/xtask/src/types.rs @@ -15,4 +15,24 @@ pub(crate) struct HunspellSource { /// Whether to run the affix rules. [`ExpandMode::Full`] unless the /// dictionary's affix table is pathological — see the enum. pub(crate) expand: ExpandMode, + /// Language-specific touch-ups to the processed list — + /// [`NO_EXTRAS`](crate::consts::NO_EXTRAS) for nearly every source. + pub(crate) extras: Extras, +} + +/// Applied in field order after the `.dic` is processed. Every entry +/// names its reason where it is listed. +pub(crate) struct Extras { + /// Attached to every form as well as kept off it: for a stems-only + /// list whose language writes its article as a prefix, so `كتاب` + /// also yields `الكتاب`. Without them the commonest form of a noun + /// is the one the dictionary lacks. + pub(crate) prefixes: &'static [&'static str], + /// `(from, to)`: every form containing `from` is kept and also + /// emitted with each `from` replaced by `to`. For a key that types + /// in one code point what the dictionary spells in two. + pub(crate) respell: &'static [(&'static str, &'static str)], + /// Forms removed last: real but rare words that would shadow a far + /// more common entry of another layout. + pub(crate) drop: &'static [&'static str], } diff --git a/xtask/src/wordlists.rs b/xtask/src/wordlists.rs index c32da526..c58f937d 100644 --- a/xtask/src/wordlists.rs +++ b/xtask/src/wordlists.rs @@ -80,7 +80,9 @@ pub(crate) fn fetch_hunspell(src_dir: &Path, wl_dir: &Path, source: &HunspellSou } let out = wl_dir.join(source.output); - if let Err(e) = process_hunspell_with_aff(&dic_path, &aff_path, &out, source.expand) { + if let Err(e) = + process_hunspell_with_aff(&dic_path, &aff_path, &out, source.expand, &source.extras) + { eprintln!(" process {} failed: {e}", dic_path.display()); return false; } @@ -116,14 +118,21 @@ pub(crate) fn encoding_of_aff(bytes: &[u8]) -> Result { // first line and hide a `SET` sitting there — pt_BR ships one. let bytes = bytes.strip_prefix(&[0xEF, 0xBB, 0xBF]).unwrap_or(bytes); - // Latin-1 view of the first 2 KB: safe on any byte, which matters - // because the encoding is what we are still looking for. - let preview: String = bytes.iter().take(2048).map(|&b| b as char).collect(); + // Latin-1 view of the whole file: safe on any byte, which matters + // because the encoding is what we are still looking for. Not just + // the head — ar.aff declares `SET` on line 360, after its `AF` table — + // and the separator may be a tab (`SET\tUTF-8`), as ar.aff writes it. + let preview: String = bytes.iter().map(|&b| b as char).collect(); let declared = preview .lines() .filter_map(|l| l.split('#').next()) - .find_map(|l| l.trim().strip_prefix("SET ")) - .map(|s| s.trim().to_uppercase()); + .find_map(|l| { + let mut words = l.split_whitespace(); + (words.next() == Some("SET")) + .then(|| words.next()) + .flatten() + }) + .map(str::to_uppercase); let normalized = declared .as_deref() @@ -212,12 +221,23 @@ pub(crate) fn process_hunspell_with_aff( aff: &Path, output: &Path, mode: ExpandMode, + extras: &Extras, ) -> Result<()> { // The .aff declares the encoding for both halves of the pair. let encoding = detect_encoding(aff)?; - let aff_text = read_hunspell_text(aff, encoding)?; - let parsed = - hunspell::Aff::parse(&aff_text).with_context(|| format!("parse {}", aff.display()))?; + // Stems-only never reads a rule, so it never needs the rules parsed: + // ar.aff uses `FLAG long` with an `AF` alias table, which the + // expander does not model and has no reason to for stems. + let parsed = match mode { + ExpandMode::Full => { + let aff_text = read_hunspell_text(aff, encoding)?; + Some( + hunspell::Aff::parse(&aff_text) + .with_context(|| format!("parse {}", aff.display()))?, + ) + } + ExpandMode::StemsOnly => None, + }; let dic_text = read_hunspell_text(dic, encoding)?; let mut words: BTreeSet = BTreeSet::new(); @@ -238,9 +258,9 @@ pub(crate) fn process_hunspell_with_aff( continue; } stem_count += 1; - let forms = match mode { - ExpandMode::Full => parsed.expand(stem, flags), - ExpandMode::StemsOnly => std::iter::once(stem.to_owned()).collect(), + let forms = match &parsed { + Some(parsed) => parsed.expand(stem, flags), + None => std::iter::once(stem.to_owned()).collect(), }; for form in forms { let lower: String = form.chars().flat_map(char::to_lowercase).collect(); @@ -253,6 +273,7 @@ pub(crate) fn process_hunspell_with_aff( } } } + apply_extras(&mut words, extras); write_sorted(output, &words)?; let name = output.file_name().and_then(|s| s.to_str()).unwrap_or("?"); @@ -296,3 +317,26 @@ pub(crate) fn http_get(url: &str) -> Result> { resp.into_reader().read_to_end(&mut buf)?; Ok(buf) } + +/// [`Extras`] in field order: prefixes, then respellings of the +/// prefixed forms too, then drops — so a dropped form cannot come back +/// through a prefix or a respelling. +fn apply_extras(words: &mut BTreeSet, extras: &Extras) { + let prefixed: Vec = extras + .prefixes + .iter() + .flat_map(|p| words.iter().map(move |w| format!("{p}{w}"))) + .collect(); + words.extend(prefixed); + for (from, to) in extras.respell { + let respelled: Vec = words + .iter() + .filter(|w| w.contains(from)) + .map(|w| w.replace(from, to)) + .collect(); + words.extend(respelled); + } + for form in extras.drop { + words.remove(*form); + } +} diff --git a/xtask/src/wordlists/tests.rs b/xtask/src/wordlists/tests.rs index 13b32c1f..a240bdf8 100644 --- a/xtask/src/wordlists/tests.rs +++ b/xtask/src/wordlists/tests.rs @@ -64,6 +64,23 @@ fn leading_bom_does_not_hide_the_set_line() { ); } +/// ar's `.aff` writes `SET\tUTF-8` on line 360, after a 333-entry `AF` +/// table — past any head-of-file preview and not matched by `"SET "`. +#[test] +fn tab_separated_set_deep_in_the_file_is_found() { + let mut text = String::from("FLAG\tlong\n"); + for i in 0..400 { + text.push_str(&format!( + "AF TaTbTcTdTeTfThTiTjTkTlTmTnToTpTqTrTsTtTuTvTxTy # {i}\n" + )); + } + text.push_str("SET\tUTF-8\nTRY\tabc\n"); + assert_eq!( + encoding_of_aff(text.as_bytes()).expect("late, tab-separated SET must be found"), + Encoding::Utf8 + ); +} + /// Both of these used to silently mean "decode as Latin-1", which is /// the behaviour that shipped broken dictionaries. They must fail. #[test] @@ -74,3 +91,19 @@ fn unknown_and_missing_encodings_are_errors() { let err = encoding_of_aff(b"TRY abc\nSFX A Y 0\n").expect_err("missing SET must fail"); assert!(err.to_string().contains("SET"), "explains itself: {err}"); } + +#[test] +fn extras_prefix_then_respell_then_drop() { + let mut words: BTreeSet = ["سلام", "كتاب", "حصي"].map(String::from).into(); + let extras = Extras { + prefixes: &["ال"], + respell: &[("لا", "ﻻ")], + drop: &["حصي"], + }; + apply_extras(&mut words, &extras); + for want in ["سلام", "السلام", "سﻻم", "السﻻم", "الكتاب", "الحصي"] + { + assert!(words.contains(want), "missing `{want}`"); + } + assert!(!words.contains("حصي"), "a dropped form must stay dropped"); +}