Sarah Hoffmann 630ad38a67 refactor variant production to use generators

2022-01-18 11:09:21 +01:00

..

sanitizers

add consistent SPDX copyright headers

2022-01-03 16:23:58 +01:00

token_analysis

refactor variant production to use generators

2022-01-18 11:09:21 +01:00

__init__.py

introduce tokenizer modules

2021-04-30 11:29:57 +02:00

base.py

add consistent SPDX copyright headers

2022-01-03 16:23:58 +01:00

factory.py

add consistent SPDX copyright headers

2022-01-03 16:23:58 +01:00

icu_rule_loader.py

add consistent SPDX copyright headers

2022-01-03 16:23:58 +01:00

icu_token_analysis.py

add consistent SPDX copyright headers

2022-01-03 16:23:58 +01:00

icu_tokenizer.py

add consistent SPDX copyright headers

2022-01-03 16:23:58 +01:00

legacy_tokenizer.py

add consistent SPDX copyright headers

2022-01-03 16:23:58 +01:00

place_sanitizer.py

add consistent SPDX copyright headers

2022-01-03 16:23:58 +01:00