Skip to content

Add Arabic, and read macOS keyboards from the OS - #83

Merged
vstrelnikof merged 4 commits into
Just-Code-NET:mainfrom
saifahmedfadl:feat/arabic
Nov 4, 2026
Merged

vstrelnikof merged 4 commits into
Just-Code-NET:mainfrom
saifahmedfadl:feat/arabic

Conversation

@saifahmedfadl

Copy link
Copy Markdown
Contributor

What & why

Adds Arabic (Arabic 101) as a bundled layout, and gives the macOS backend a describe_keymaps so the keyboard actually installed outranks the bundled table, as on Windows.

The second part matters because Apple ships three Arabic keyboards (Arabic, Arabic - PC, Arabic - North Africa) that differ on most of the bottom row, and all three map to one LayoutId. On my Mac (Arabic - North Africa) the log shows adopted the OS keymap … layout=ar … keys=47 replaced=41.

Trade-offs:

  • Lam-alef keys. B types لا, which is two code points, and a key entry is one char. The table carries the presentation forms (ﻻ U+FEFB, plus ﻷ ﻹ ﻵ on the shifted keys), and the macOS reader folds the OS's two-code-point output the same way. Leaving the key out was worse: an unmapped key is dropped from a translation, so gdb read as لي, a real word, and shell_vocabulary_shadows_no_other_layouts_word caught it. Script::of now counts the presentation-form blocks as Arabic.
  • Dictionary. Ayaspell's .aff uses FLAG long with an AF alias table, so ar is StemsOnly, like Hebrew. Stems carry no article, though, and ال starts a large share of running text. So HunspellSource gains an Extras field: prefixes (ال وال بال فال كال لل), respellings (لا → ﻻ, so hgsbl still finds السلام), and a drop list (حصي, which pwd types as). Every other source uses NO_EXTRAS. The result is 2.5 M forms, 6.3 MB gz, mid-pack among the bundled lists. A bare one-letter clitic (وكتاب) still misses and falls to plausibility.
  • Vowels. With an empty vowel set, plausibility scores every word in that script 0.0: one long consonant cluster. derive_vowels("ar") now returns the written long vowels (alef forms, waw, yeh, alef maksura, the lam-alef forms). Hebrew has the same empty set today; I left it alone.
  • Tatweel (ـ, alphabetic) is kept off Shift+- in both the TOML and the macOS reader, so _ never reads as a letter and the snake_case heuristic stays on.
  • xtask: encoding_of_aff now finds a tab-separated SET on line 360 of ar.aff, and a stems-only dictionary no longer parses rules it never uses.

New test arabic_and_english_typed_in_the_wrong_layout_are_switched runs the real detectors both ways. السلام عليكم كتاب مدرسة typed on en-US switch to ar, including السلام typed with B. hello world table school typed on ar switch to en-US. Neither word changes in its own layout.

Known gap (Windows), follow-up: an Arabic HKL resolves to ar-EG / ar-SA / …, not ar, so the bundled layout is not matched there yet. This needs a language-subtag fallback in the id mapping. I did not want to guess at it without a Windows box to test on. Linux maps ara, and the MSI ships the files.

Platforms touched

macOS (layout backend), all (data, poltertype-detect script ranges, xtask), Windows (MSI file entries only).

Checklist

  • cargo fmt --all and cargo clippy --workspace --all-targets --all-features -- -D warnings are clean
  • cargo test --workspace passes
  • Docs that describe the changed behaviour are updated in the same PR (README, CHANGELOG, data/wordlists/CREDITS.md)
  • Platform-specific code stays inside the per-OS crates
  • When touching OS APIs: UCKeyTranslate, TISGetInputSourceProperty / kTISPropertyUnicodeKeyLayoutData, LMGetKbdType

describe_keymaps on macOS walks the character block of every selectable
keyboard through UCKeyTranslate, the counterpart of the Windows
ToUnicodeEx walk, so a variant whose keys differ from the bundled table
(Apple's Arabic against Arabic 101) is corrected on the keys the user
has. The current keyboard comes first, as the DB keeps the first of a
language. Lam-alef keys fold to their presentation form; tatweel is
left out so `_` never reads as a letter.
HunspellSource gains Extras: prefixes attached to every form, respellings
for a key that types one code point where the dictionary spells two,
and forms dropped last. Every existing source uses NO_EXTRAS.

encoding_of_aff now finds a tab-separated SET past the head of the file,
and a stems-only dictionary no longer parses affix rules it never uses.
ar.toml read off macOS ArabicPC, a 2.5 M-form Ayaspell wordlist (stems,
plus article forms and the B-key lam-alef spelling), a stop list checked
against en-US, Arabic's long vowels for plausibility, and presentation
forms counted as Arabic script. Mapped on macOS (all Apple Arabic
variants), Linux (ara) and in the MSI.
Arabic and Hebrew leave short vowels unwritten, so a correctly typed word
with no long vowel (بسرعه) scored 0.0 as one consonant cluster while its
en-US rendering (fsnui) scored 0.75, and plausibility switched it away.
Text wholly in the current layout's abjad is now kept by plausibility;
leaving it takes a dictionary hit. Found in daily use on macOS.
@vstrelnikof
vstrelnikof merged commit 84f6882 into Just-Code-NET:main Nov 4, 2026
5 checks passed
@vstrelnikof

Copy link
Copy Markdown
Member

Merged and released in 0.39.0, thank you. Your commits are in as they were. On top I fixed what live typing showed: B puts two code points on screen, so words were erased one short (table left فtable, on Linux too); ar.toml now follows xkb's ara; Windows maps every ar-* locale to ar; and English outside the dictionary can leave Arabic and Hebrew again when the other dictionary follows much more of the word.

Could you try table on your Arabic keyboard on the Mac and say whether it comes back clean? Two Backspaces for a lam-alef key is measured on Linux and only assumed on macOS.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants