Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 28 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,34 @@ All notable changes to PolterType are recorded here. The format is
loosely based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
and the project follows [Semantic Versioning](https://semver.org/).

## [Unreleased]

### Added

- **Arabic.** A bundled `ar` layout (Arabic 101, read off the OS rather
than transcribed), a 2.5 M-form dictionary from Ayaspell's stems
with the article forms added, Arabic's long vowels for the
plausibility detector, and a short
stop-word list checked so that none of its words is what an English
stop word types as. macOS maps `Arabic`, `Arabic - PC`,
`Arabic - QWERTY`, `Arabic - AZERTY` and `Arabic - North Africa` to
it; Linux maps `ara`.
- **macOS reads your keyboards from the OS.** The layout backend now
answers `describe_keymaps` with `UCKeyTranslate`, as the Windows one
does with `ToUnicodeEx`, so a variant whose keys differ from the
bundled table — Apple's own Arabic against Arabic 101, Russian
Phonetic against Russian — is corrected on the keys you actually
have.

### Fixed

- **Arabic presentation forms count as Arabic script.** `ﻻ` and the
rest of U+FB50–FDFF and U+FE70–FEFF read as "other script" and
dragged a word's script fit down.
- **`cargo xtask wordlists fetch` finds a `SET` anywhere in the `.aff`,
tab-separated or not,** and no longer parses rules it will not use
for a stems-only dictionary.

## [0.38.2] — hands off first

### Fixed
Expand Down
15 changes: 10 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -386,22 +386,22 @@ them on the **Commands** pane in Settings.

## Languages

Fifteen layouts ship with the app, each with a full dictionary:
Sixteen layouts ship with the app, each with a full dictionary:

**English (US) · Ukrainian · Russian · German · Spanish · French ·
Polish · Czech · Greek · Hebrew · Turkish · Bulgarian · Italian ·
Portuguese (PT) · Portuguese (BR)**
Portuguese (PT) · Portuguese (BR) · Arabic**

PolterType only loads the ones your OS actually has enabled, so
bundling fifteen costs a two-keyboard user nothing at runtime.
bundling sixteen costs a two-keyboard user nothing at runtime.

**On Windows the bundled table is only a starting point.** A Windows
layout is named after a *language*, but a language is not a keyboard —
Bulgarian alone ships in three genuinely different arrangements. Since
v0.14.3 PolterType asks Windows what each of your installed keyboards
actually produces and uses that answer, so variants and even custom
layouts we have never heard of come out right. Linux and macOS read
the bundled tables, and a TOML of your own in
layouts we have never heard of come out right; macOS does the same
through `UCKeyTranslate`. Linux reads the bundled tables, and a TOML of your own in
`<config-dir>/poltertype/layouts/` still outranks everything,
everywhere.

Expand All @@ -418,6 +418,11 @@ Two are worth a footnote rather than a surprise:
because expanding its clitic prefixes yields 60 million of them.
Hebrew shares its script with nothing else bundled, so detection
leans on that and the dictionary refines it.
- **Arabic** follows Hebrew for the same reason — its article and
clitic particles are prefixes — and adds the article forms (`ال`,
`بال`, `لل`, …) back to the list. The lam-alef keys (`لا` on `B`) type
two code points, and a key here is one character, so they carry the
single-code-point form `ﻻ`, which looks the same.

**Yours isn't here?** Adding a language is one TOML file and one
wordlist — no Rust required. The walkthrough is
Expand Down
1 change: 1 addition & 0 deletions crates/poltertype-core/build.rs
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,7 @@ const LAYOUTS: &[(&str, &str)] = &[
("it_it", "it-IT"),
("pt_pt", "pt-PT"),
("pt_br", "pt-BR"),
("ar", "ar"),
];

fn main() {
Expand Down
78 changes: 78 additions & 0 deletions crates/poltertype-core/src/engine/tests.rs
Original file line number Diff line number Diff line change
Expand Up @@ -754,6 +754,84 @@ mod engine_integration_tests {
);
}

/// Both directions of the Arabic ↔ English mix-up through the real
/// pipeline, and neither word disturbed in its own layout. Renders
/// each word through both tables the way the engine does, then asks
/// the detectors.
#[test]
fn arabic_and_english_typed_in_the_wrong_layout_are_switched() {
use crate::layouts::LayoutDb;
let db = LayoutDb::load_embedded();
let (en, ar) = (LayoutId::from("en-US"), LayoutId::from("ar"));
let en_map = db.get(&en).expect("en-US");
let ar_map = db.get(&ar).expect("ar");
let render = |typed: &str| -> String {
typed
.chars()
.map(|ch| {
let (sc, shift) = en_us_key(en_map, ch);
let (plain, shifted) = ar_map.keys[&sc];
if shift {
shifted.unwrap_or(plain)
} else {
plain
}
})
.collect()
};
let judge = |current: &LayoutId, typed: &str| -> Option<LayoutId> {
let candidates = vec![(en.clone(), typed.to_owned()), (ar.clone(), render(typed))];
let ctx = poltertype_detect::DetectionContext {
current_layout: current,
candidates: &candidates,
recent_context: "",
};
for d in real_detectors() {
match d.judge(&ctx) {
Verdict::NoOpinion => continue,
Verdict::Keep { .. } => return None,
Verdict::Switch(v) => return Some(v.best_layout),
}
}
None
};
// السلام عليكم, كتاب, مدرسة on Arabic 101 — and السلام again
// with its lam-alef typed on `B`, which renders as `ﻻ`.
for typed in ["hgsghl", "ugd;l", ";jhf", "l]vsm", "hgsbl"] {
assert_eq!(
judge(&en, typed),
Some(ar.clone()),
"{typed:?} -> {}",
render(typed)
);
assert_eq!(
judge(&ar, typed),
None,
"{} is Arabic typed in Arabic",
render(typed)
);
}
// بسرعه and كتب carry no long vowel: one consonant cluster to the
// shape score, while their en-US renderings look like words.
// Typed in Arabic they must stay Arabic.
for typed in ["fsvui", ";jf"] {
assert_eq!(
judge(&ar, typed),
None,
"{} is Arabic typed in Arabic",
render(typed)
);
}
for typed in ["hello", "world", "table", "school"] {
assert_eq!(judge(&en, typed), None, "{typed:?} is English");
assert_eq!(
judge(&ar, typed),
Some(en.clone()),
"{typed:?} typed in Arabic"
);
}
}

/// Issue #33, the residue on 0.21.0: `auto-switch ` typed in
/// en-US came back as `ФГЕЩ-ЫЦШЕСР `. Both halves of that need
/// explaining, and this pins the half that needs no Caps Lock —
Expand Down
2 changes: 1 addition & 1 deletion crates/poltertype-core/src/layouts/consts.rs
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@
/// "missing TOML" warning at startup, never silently.
pub const BUNDLED_LAYOUT_STEMS: &[&str] = &[
"en_us", "uk_ua", "ru_ru", "de_de", "es_es", "fr_fr", "pl_pl", "cs_cz", "el_gr", "he_il",
"tr_tr", "bg_bg", "it_it", "pt_pt", "pt_br",
"tr_tr", "bg_bg", "it_it", "pt_pt", "pt_br", "ar",
];

/// Fewest keys an [`poltertype_types::OsKeymap`] must carry before it
Expand Down
5 changes: 5 additions & 0 deletions crates/poltertype-core/src/layouts/helpers.rs
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,11 @@ pub fn derive_vowels(id: &LayoutId, script: Script) -> Vec<char> {
"bg-BG" => "аеиоуъюя".chars().collect(),
"it-IT" => "aeiouàèéìíîòóùú".chars().collect(),
"pt-PT" | "pt-BR" => "aeiouáàâãéêíóôõúü".chars().collect(),
// Arabic writes only its long vowels: alef in every hamza and
// madda form, waw, yeh, alef maksura, and the lam-alef keys.
// An empty set scored every Arabic word as one long consonant
// cluster — 0.0 — so plausibility could never pick Arabic.
"ar" => "اأإآويىﻻﻵﻷﻹ".chars().collect(),
_ => match script {
Script::Latin => "aeiouy".chars().collect(),
Script::Cyrillic => "аеиіоуюяєїыэё".chars().collect(),
Expand Down
6 changes: 5 additions & 1 deletion crates/poltertype-core/src/layouts/tests.rs
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ fn embedded_layouts_load() {
let db = LayoutDb::load_embedded();
for id in [
"en-US", "uk-UA", "ru-RU", "de-DE", "es-ES", "fr-FR", "pl-PL", "cs-CZ", "el-GR", "he-IL",
"tr-TR", "bg-BG", "it-IT", "pt-PT", "pt-BR",
"tr-TR", "bg-BG", "it-IT", "pt-PT", "pt-BR", "ar",
] {
assert!(
db.get(&LayoutId::from(id)).is_some(),
Expand Down Expand Up @@ -112,6 +112,9 @@ fn new_languages_translate_distinctive_keys() {
// Hebrew is unicase — the letter positions are RTL glyphs.
("he-IL", 0x1E, false, 'ש'),
("he-IL", 0x14, false, 'א'),
// Arabic is unicase too; `B` types lam-alef as one code point.
("ar", 0x1E, false, 'ش'),
("ar", 0x30, false, '\u{FEFB}'),
// Turkish's dotless/dotted i pair, the one that bites.
("tr-TR", 0x17, false, 'ı'),
("tr-TR", 0x17, true, 'I'),
Expand Down Expand Up @@ -233,6 +236,7 @@ fn every_bundled_dictionary_knows_its_own_words() {
("cs-CZ", &["ahoj", "slovo", "dobrý", "protože", "děkuji"]),
("el-GR", &["λέξη", "είναι", "καλημέρα", "ευχαριστώ"]),
("he-IL", &["שלום", "מילה", "ספר", "תודה"]),
("ar", &["كتاب", "مدرسة", "شكرا", "جميل"]),
(
"tr-TR",
&["kitap", "merhaba", "kelime", "güzel", "teşekkür"],
Expand Down
4 changes: 3 additions & 1 deletion crates/poltertype-detect/src/enums.rs
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,9 @@ impl Script {
0x0370..=0x03FF => Self::Greek,
0x0530..=0x058F => Self::Armenian,
0x0590..=0x05FF => Self::Hebrew,
0x0600..=0x06FF => Self::Arabic,
// The presentation forms count too: the lam-alef keys of an
// Arabic keyboard type U+FEFB and its siblings.
0x0600..=0x06FF | 0x0750..=0x077F | 0xFB50..=0xFDFF | 0xFE70..=0xFEFF => Self::Arabic,
_ => Self::Other,
}
}
Expand Down
29 changes: 29 additions & 0 deletions crates/poltertype-detect/src/plausibility.rs
Original file line number Diff line number Diff line change
Expand Up @@ -98,6 +98,22 @@ impl WordPlausibilityDetector {
None
}

/// Whether `text` is wholly letters of `layout`'s own abjad. Arabic
/// and Hebrew leave short vowels unwritten, so a correctly typed word
/// reads as one consonant cluster (`بسرعه` scores 0.0) while its
/// Latin rendering lands vowels (`fsnui` scores 0.75) and wins.
/// Shape cannot tell those apart; only the dictionary can.
fn abjad_in_own_script(&self, layout: &LayoutId, text: &str) -> bool {
let Some(prof) = self.profiles.get(layout) else {
return false;
};
let mut letters = text.chars().filter(|c| c.is_alphabetic()).peekable();
matches!(prof.script, Script::Arabic | Script::Hebrew)
&& non_word_char_count(text) == 0
&& letters.peek().is_some()
&& letters.all(|c| Script::of(c) == prof.script)
}

/// The plain word-shape score, with no compound handling.
fn score(prof: &LayoutProfile, text: &str) -> f32 {
let letters: Vec<char> = text
Expand Down Expand Up @@ -216,6 +232,19 @@ impl Detector for WordPlausibilityDetector {
};
}

// Leaving an abjad is the dictionary's call: an English word typed
// on the Arabic keys is a dictionary hit there, and anything less
// is safer kept as typed than replaced on shape alone.
if self.abjad_in_own_script(ctx.current_layout, current_text) {
return Verdict::Keep {
reason: format!(
"current {} is in {}'s own abjad; left to the dictionary",
logsafe::redact_word(current_text),
ctx.current_layout
),
};
}

if current_fit >= self.keep_threshold {
return Verdict::Keep {
reason: format!(
Expand Down
1 change: 1 addition & 0 deletions crates/poltertype-layout/src/linux/shared.rs
Original file line number Diff line number Diff line change
Expand Up @@ -100,6 +100,7 @@ pub fn bcp47_to_xkb(bcp: &str) -> Option<&'static str> {
"hy-AM" => "am",
"ka-GE" => "ge",
"he-IL" => "il",
"ar" => "ara",
"ja-JP" => "jp",
"ko-KR" => "kr",
_ => return None,
Expand Down
39 changes: 28 additions & 11 deletions crates/poltertype-layout/src/macos.rs
Original file line number Diff line number Diff line change
Expand Up @@ -16,9 +16,11 @@ use core_foundation::base::{CFRelease, CFTypeRef, OSStatus, TCFType};
use core_foundation::dictionary::{CFDictionary, CFDictionaryRef};
use core_foundation::string::{CFString, CFStringRef};

use crate::{LayoutError, LayoutId, LayoutSwitcher};
use crate::{LayoutError, LayoutId, LayoutSwitcher, OsKeymap};

mod consts;
mod ids;
mod keymap;
use ids::{bcp47_to_tis_id, source_id_string, source_id_to_layout_id, tis_id_to_bcp47};

// ─── Carbon FFI ──────────────────────────────────────────────────────
Expand Down Expand Up @@ -152,6 +154,10 @@ impl LayoutSwitcher for MacosLayoutSwitcher {
fn backend_name(&self) -> &'static str {
"macos-tis"
}

fn describe_keymaps(&self) -> Result<Vec<OsKeymap>, LayoutError> {
run_on_main(keymap::describe_all)
}
}

fn current_impl() -> Result<LayoutId, LayoutError> {
Expand All @@ -170,8 +176,9 @@ fn current_impl() -> Result<LayoutId, LayoutError> {
}
}

fn list_active_impl() -> Result<Vec<LayoutId>, LayoutError> {
// Filter for keyboard sources that the user can select.
/// The keyboard sources the user can select — what `list_active` names
/// and `describe_keymaps` describes. `None` when TIS returns no list.
fn selectable_keyboards() -> Option<CFArray<CFTypeRef>> {
let filter = unsafe {
let category_key = CFString::wrap_under_get_rule(kTISPropertyInputSourceCategory);
let category_val = CFString::wrap_under_get_rule(kTISCategoryKeyboardInputSource);
Expand All @@ -189,17 +196,24 @@ fn list_active_impl() -> Result<Vec<LayoutId>, LayoutError> {
unsafe {
let arr_ref = TISCreateInputSourceList(filter.as_concrete_TypeRef(), false);
if arr_ref.is_null() {
return Ok(Vec::new());
return None;
}
let arr: CFArray<CFTypeRef> = CFArray::wrap_under_create_rule(arr_ref);
let mut out = Vec::with_capacity(arr.len() as usize);
for src in arr.iter() {
if let Some(id) = source_id_to_layout_id(*src) {
out.push(id);
}
Some(CFArray::wrap_under_create_rule(arr_ref))
}
}

fn list_active_impl() -> Result<Vec<LayoutId>, LayoutError> {
let Some(arr) = selectable_keyboards() else {
return Ok(Vec::new());
};
let mut out = Vec::with_capacity(arr.len() as usize);
for src in arr.iter() {
// Safety: `src` lives as long as `arr`.
if let Some(id) = unsafe { source_id_to_layout_id(*src) } {
out.push(id);
}
Ok(out)
}
Ok(out)
}

fn switch_to_impl(id: &LayoutId) -> Result<(), LayoutError> {
Expand Down Expand Up @@ -236,3 +250,6 @@ fn switch_to_impl(id: &LayoutId) -> Result<(), LayoutError> {

Err(LayoutError::NotActive(id.clone()))
}

#[cfg(test)]
mod tests;
Loading
Loading