Sarah Hoffmann e85f7e7aa9 fix subsequent replacements ...

Two replacement words directly following each other did not
work as expected because each expects a space at the
beginning/end while there was only one space available.

Also forbit composing a word after a space was added in the
end by a previous replacement.

2021-07-04 10:28:28 +02:00

..

__init__.py

introduce tokenizer modules

2021-04-30 11:29:57 +02:00

factory.py

do not hide errors when importing tokenizer

2021-05-18 16:28:21 +02:00

icu_name_processor.py

fix subsequent replacements

2021-07-04 10:28:28 +02:00

icu_rule_loader.py

switch to a more flexible variant description format

2021-07-04 10:28:20 +02:00

icu_variants.py

leave ICU variant properties empty for now

2021-07-04 10:28:20 +02:00

legacy_icu_tokenizer.py

only consider partials in multi-words for initial count

2021-07-04 10:28:20 +02:00

legacy_tokenizer.py

adapt tests for ICU tokenizer

2021-07-04 10:28:20 +02:00