Sarah Hoffmann 6ba00e6aee icu tokenizer: move transliteration rules in separate file ...

The tokenizer configuration has become difficult to handle
due to the additional manual transliteration rules. Allow
to have a separate rule file that is given to the ICU library
as is.

2021-07-04 10:28:20 +02:00

..

__init__.py

introduce tokenizer modules

2021-04-30 11:29:57 +02:00

factory.py

do not hide errors when importing tokenizer

2021-05-18 16:28:21 +02:00

legacy_icu_tokenizer.py

icu tokenizer: move transliteration rules in separate file

2021-07-04 10:28:20 +02:00

legacy_tokenizer.py

fix insertion of special terms and countries into word table

2021-06-02 20:22:39 +02:00