Repository navigation
Romanian Normalization - #34
Conversation
Add Romanian (`ro`) as a self-contained language: - RomanianNumberNormalizer: spelled-out cardinals to digits (units, teens, tens, "și" compounds, hundreds, mie/milion/miliard with the "de" Romanian uses from 20 up), the colloquial contracted forms (cinșpe = 15, douăj = 20), ordinals after "al"/"a" and digit ordinals (al 3-lea, a 3-a). "un"/"o" count as 1 only before sută/mie/milion/miliard, so articles stay words. Words match with or without diacritics, and the legacy cedilla letters ş/ţ are read as ș/ț. - LanguageConfig: decimal comma with "virgula" (folded, since the decimal-word steps run after remove_diacritics), thousands dot, symbols and currencies, digit and number words. - Filler words only where their diacritic-folded form cannot be a Romanian word: "ăă" becomes "aa" and is removed, but "ă" (→ "a") and "îî" (→ "ii", the pronoun "îi") are kept. - Replacements: okay/oke/okei/okey → ok. - Unit tests for the number normalizer and operators, a gladia-3 e2e CSV, README row.
- Drop the "de" that links a number of at least 20 (or ending in 00) to its noun, so
"20°", "20 de grade" and "douăzeci de grade" all normalize to "20 grade" (and
"300 de lei" / "trei sute de lei" to "300 lei"). "de" after smaller numbers and in
"de la" ("from") is kept.
- Recognize the informal teens of everyday speech ("cincișpe"/"cincispe" = 15) next to
the standard and most reduced forms.
- Read "și" compounds written as one word ("douăzecișicinci" = 25).
- E2e cases rewritten as natural sentences (battery level, discounts, prices, temperature).
"1.200 €", "1.200€" and "€1.200" become "1200 euro", matching the spoken "o mie două sute de euro" (the shared pipeline drops a symbol written after the amount with a space). As in the Danish normalizer, the Romanian number normalizer rewrites the config's currency symbols itself.
|
Thanks a lot for this contribution @eek, the Romanian support looks great and is really appreciated! One thing is blocking the merge for now: the Lint commit messages check fails on the last commit ( Our commitlint config requires the subject to start with a lowercase letter. Since it's the latest commit, an amend is enough, for example: git commit --amend -m "feat: romanian clock times, halves, units and currency codes" -m "$(git log -1 --format=%b)"
git push --force-with-leaseOnce the check is green we can move forward. Thanks again! |
| def at(k: int) -> str: | ||
| return rest[k] if k < len(rest) else "" | ||
|
|
||
| if at(0) in _HALF: |
There was a problem hiding this comment.
False positive: percentages and amounts are read as clock times
After la / e, any X și jumătate / X și un sfert becomes H:MM, even when an amount follows:
Inflația a crescut la trei și jumătate la sută. → inflatia a crescut la 3:30 la suta
Inflația e trei și jumătate la sută. → inflatia e 3:30 la suta
Rata a scăzut la doi și un sfert la sută. → rata a scazut la 2:15 la suta
Inflația a urcat la 3,5%. → inflatia a urcat la 3 virgula 5 la suta
So a spelled-out reference and a digit hypothesis (or the reverse) no longer match, and this pattern is frequent in news-style text such as FLEURS. Could you skip the time reading when the phrase is followed by la sută / %, a unit, or mii / milioane, and add these sentences to ro.csv?
| for before, after, word in self._currency: | ||
| text = before.sub(rf"\1 {word}", text) | ||
| text = after.sub(rf"\1 {word}", text) |
There was a problem hiding this comment.
Currency symbol binds to the next number
The symbol-before rule runs first, so a symbol written after an amount gets attached to the following number instead, and the digits then collapse:
Am cumpărat 5 € 2 kilograme de mere. → am cumparat 52 euro kg de mere
Since Romanian writes the symbol after the amount, applying the after pattern first (or not matching before when the symbol directly follows a number) should fix it. A test row for this case would be great too.
karamouche
left a comment
karamouche
left a comment
There was a problem hiding this comment.
Thanks for your contribution! Here is some feedback flagged by AI. I don't speak Romanian, so I don't know if it's all relevant, but I'm flagging it so you can help.
Details are in the inline comments, plus the commitlint fix mentioned above.
- Spoken clock times after "la"/"ora"/"e" become H:MM before the time-colon protection, so they match written times: "la opt și jumătate" / "la opt jumate" / "la ora 8.30" → 8:30, "opt și(-un) sfert" → 8:15, "opt fără un sfert" and "șapte patruzeci și cinci" → 7:45, "opt fără zece" → 7:50 (time_normalizer.py, via normalize_numeric_time_formats). - A half after an amount is its decimal: "trei kile jumate", "trei kilograme și jumătate" and "3,5 kg" all become "3 virgula 5 kg"; "jumătate de ..." is left alone. - Units: kil/kile/kilogram(e) → kg, kilometru/kilometri → km. - Currency codes and acronyms stay words (expand_all_caps_letter_by_letter=False): "RON" is "ron", not "r o n", and stays distinct from "lei". - Informal numbers: douășpe/doușpe (12, feminine), fused tens-and-five (douășcinci, treișcinci, patrușcinci).
…amount - A time-like phrase followed by an amount is a quantity, whatever its cue: "la trei și jumătate la sută" is "3 virgula 5 la suta" like "3,5%", not 3:30; also before "%", "procent(e)", "mii"/"milioane"/"miliarde", currency words and units (review on gladiaio#34). - "și un sfert" after an amount is its decimal, like a half: "doi și un sfert la sută" and "două ore și un sfert" become "2 virgula 25 ...", like "2,25"; "un sfert de oră" stays. - A currency symbol after an amount is read before one before it, as Romanian writes it, so "5 € 2 kilograme" is "5 euro 2 kg" instead of "52 euro kg" (review on gladiaio#34). - Unit tests and gladia-3 ro.csv rows for these cases.
- Times: a dotted date ("până la 15.03.2024") or a dotted amount ("la 8.30 lei") is not a
clock time; nor is a phrase whose unit follows "de" ("la zece douăzeci de lei") or a range
of amounts ("de la trei și jumătate la cinci la sută"). "este" and "după" are cues like
"e"/"la", and after "ora" any minutes may follow "și" ("la ora opt și cincisprezece").
- Halves and quarters: a "de" after the noun stays the noun's ("trei kilograme și jumătate
de mere" = "3,5 kilograme de mere"); "un"/"o" are 1 before a measure, whose decimal takes
the plural ("o oră și jumătate" = "1,5 ore"); without "și" only after a measure ("cei doi
vor jumătate" is left alone); after a multiplier a decimal of it ("două milioane și
jumătate" = "2,5 milioane", not 2000000,5).
- Currency: read once spelled numbers are digits ("cinci € 2 kilograme" → "5 euro 2"), a
symbol after an amount never taking the next amount's own ("€50 €100" → "50 euro 100
euro", which the previous fix broke), "20 de €" → "20 euro", and "€12,50" keeps its decimal.
- Ordinals: "a opta" and those built on any cardinal ("al unsprezecelea", "a douăzecea",
"al douăzeci și cincilea"); "întâi" before a month is 1 ("întâi decembrie").
- The "de" of a preposition or discourse marker after a number stays ("douăzeci de pe
listă", "în 2024 de exemplu").
- Roman numerals only in capitals (roman_numerals_uppercase_only): "vii" (you come), "vi"
and "ii" without diacritics are words; "Capitolul VII" is still 7.
- The decimal word comes from the language config.
News drops the unit: "Inflația a crescut la trei și jumătate" is 3,5, like "a crescut la
3,5", not 3:30. A "la" after a verb of change (crescut, scăzut, urcat, coborât, redus,
majorat, ...), also over "de"/"până" and a range's numbers ("a scăzut de la patru la trei și
jumătate"), is not read as a time; "a ajuns la opt și jumătate" and "de la opt la nouă și
jumătate" still are.
66fc944 to
bbe86ac
Compare
|
Thanks for the review! All three points are fixed:
A second automated review found more, now fixed: dotted dates after "la" (15.03.2024) were read as times, "două milioane și jumătate" became 2000000,5, "€50 €100", "20 de €", spelled numbers before €, "o oră și jumătate", ordinals like "a opta" and "al douăzecilea", and lowercase Roman numerals ("vii", "you come", was becoming 7; now capitals only, as for Norwegian). All cases are in ro.csv and the unit tests. |
|
It looks legit, thanks for the contribution, don't hesitate to improve the romanian normalizer in the future :) |
What does this PR do?
Adds Romanian (
ro) normalization: spelled-out numbers to digits, Romanian fillers, symbols,currencies, clock times and decimal conventions, so Romanian STT output and references compare fairly.
Type of change
Checklist
New language
normalization/languages/romanian/withoperators.py,replacements.py,__init__.pyreplacements.py(not hardcoded inoperators.py)LanguageConfigis filled in with the language's data (separators, currency words, digit words, …)LanguageOperators— onlyexpand_written_numbersandget_word_replacementsare overridden@register_languageand imported innormalization/languages/__init__.pytests/unit/languages/tests/e2e/files/gladia-3/ro.csvNotes
text2numhas no Romanian, sonumber_normalizer.pyparses cardinals: units, teens, tens,"și" compounds (douăzeci și cinci = 25), hundreds, mie/milion/miliard with the "de" Romanian
uses from 20 up (douăzeci de mii = 20000), colloquial contracted forms (cinșpe = 15,
douăj = 20), and ordinals after "al"/"a" (al treilea = al 3-lea = al 3).
removed, but "ă" (→ "a") and "îî" (→ "ii", the pronoun "îi") are kept.
decimal_wordis the folded "virgula", because the decimal-word steps run afterremove_diacritics; this makes "3,5" and "trei virgulă cinci" agree."o sută de lei"); written forms ("20°", "20 grade") have none, so that "de" is dropped after
such numbers. It is kept after smaller numbers and in "de la" ("from").
forms ("cinșpe") are recognized, as are "și" compounds written as one word ("douăzecișicinci").
matching "o mie două sute de euro". The shared pipeline drops a symbol written after the
amount with a space, so, as in the Danish normalizer, the Romanian number normalizer
rewrites the config's currency symbols itself.
"opt fără un sfert", "șapte patruzeci și cinci", and "8.30" written with a dot) become H:MM.
They are converted in
normalize_numeric_time_formatsbecause it runs before the time-colonprotection; a dedicated hook for spoken times may be preferable in a future preset.
"3,5 kg"); kilogram/kilometre spellings map to kg/km.
expand_all_caps_letter_by_letter=False): "RON" is not "lei".How was this tested?