Repository navigation
feat(ar): add Arabic normalization - #36
Draft
karamouche wants to merge 10 commits into
Draft
karamouche wants to merge 10 commits into
karamouche wants to merge 10 commits into
Conversation
Adds an optional character_replacements field to LanguageConfig and a step that applies it first in gladia-3 text_pre. The step is a no-op for every language that leaves the field as None, so existing outputs are unchanged.
Registers the 'ar' language with symbol and currency words, Arabic filler words and an empty word replacement table.
Vowel marks were turned into spaces by remove_symbols, splitting vocalized words into single letters (مَرْحَبًا -> م ر ح ب ا). They are now deleted up front, together with the typographic tatweel.
Folds hamzated/madda/wasla alef to bare alef, alef maqsura to yeh and ta marbuta to heh, plus Persian yeh/keheh lookalikes, so spelling variants of the same word no longer count as WER errors.
Maps Arabic-Indic and Eastern Arabic-Indic digits to 0-9 and the Arabic decimal separator, thousands separator and percent sign to their ASCII counterparts. Previously ١٢٫٥ collapsed to ١٢٥.
Once alef is folded, the plural "أمم" (nations) normalizes to "امم" and was deleted as a hesitation. "اه" is also the dialectal "yes" in Egyptian and Levantine Arabic. Only the elongated forms (اهه, اممم, ممم) are fillers now.
…ized form remove_diacritics already folded ؤ to و and ئ to ي through NFKD, but only as a side effect, so symbol words such as "زائد" and "بالمئه" did not match what the pipeline outputs and replacement keys with hamza seats could never match. The folding is now part of character_replacements, the config words are written in their final form, and a test guards that invariant.
Adds ArabicNumberNormalizer behind expand_written_numbers, so references and hypotheses that write the same number as words or digits normalize alike (خمسة وعشرون and ٢٥ both give 25). Covers MSA cardinals up to billions with gender and case variants, two-word teens, units-before-tens compounds chained with و, glued and separate hundreds, ألف/آلاف/مليون/ملايين groups, digits followed by a scale word, and common Egyptian and Levantine spellings. و is only consumed where it builds a compound, so ثلاثة وأربعة stays "3 و4" like ٣ و٤. Standalone واحد/أحد stay words, as English keeps "one". digit_words enables digit-by-digit readings. Ordinals, fractions (نص, ربع) and dual nouns are out of scope.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Adds Arabic (
ar) normalization: it strips vowel marks (harakat) and tatweel, unifies alef / alef maqsura / ta marbuta variants, and converts Arabic-Indic digits and separators. This is driven by a new genericapply_character_replacementsstep. Previously Arabic fell back todefault, which split vocalized words into single letters (مَرْحَبًا→م ر ح ب ا) and dropped the Arabic decimal separator (١٢٫٥→١٢٥).Not covered yet: spelled-out Arabic numbers (
ثلاثة→3), and the word replacement list is still empty.Type of change
Checklist
New language
normalization/languages/{lang}/withoperators.py,replacements.py,__init__.pyreplacements.py(not hardcoded inoperators.py)LanguageConfigis filled in with the language's data (separators, currency words, digit words, …)LanguageOperators— only override methods where the logic changes, not just the data@register_languageand imported innormalization/languages/__init__.pytests/unit/languages/tests/e2e/files/{preset}/{lang}.csv(e.g.tests/e2e/files/gladia-3/fr.csv)New step
nameclass attribute set (this is the key used in YAML presets)@register_stepand imported insteps/text/__init__.pyorsteps/word/__init__.pyoperators.config.*insteadsteps/text/placeholders.pyandpipeline/base.py'svalidate()is updated (N/A)tests/unit/steps/uv run scripts/generate_step_docs.pyHow was this tested?
1093 passed. New tests:
tests/unit/steps/text/apply_character_replacements_test.py,tests/unit/languages/arabic_operators_test.py,tests/e2e/files/gladia-3/ar.csv.