Sarah Hoffmann 8377528952 new word table layout for icu tokenizer ...

The table now directly reflects the different token types.
Extra information is saved in a json structure that may be
dynamically extended in the future without affecting the
table layout.

2021-07-28 11:31:47 +02:00

..

__init__.py

introduce tokenizer modules

2021-04-30 11:29:57 +02:00

factory.py

do not hide errors when importing tokenizer

2021-05-18 16:28:21 +02:00

icu_name_processor.py

split up variant computation for better readability

2021-07-12 14:43:50 +02:00

icu_rule_loader.py

switch to a more flexible variant description format

2021-07-04 10:28:20 +02:00

icu_variants.py

remove default parameter for namedtuple

2021-07-06 22:57:42 +02:00

legacy_icu_tokenizer.py

new word table layout for icu tokenizer

2021-07-28 11:31:47 +02:00

legacy_tokenizer.py

add helper function for execute_values

2021-07-12 21:08:20 +02:00