Sarah Hoffmann 6c6bbe5747 add type annotations for ICU tokenizer

2022-07-18 09:47:57 +02:00

..

sanitizers

add type hints for sanitizers

2022-07-18 09:47:57 +02:00

token_analysis

add type annotations for token analysis

2022-07-18 09:47:57 +02:00

__init__.py

introduce tokenizer modules

2021-04-30 11:29:57 +02:00

base.py

add type annotations for legacy tokenizer

2022-07-18 09:47:57 +02:00

factory.py

add type annotations to ICU tokenizer helper modules

2022-07-18 09:47:57 +02:00

icu_rule_loader.py

add type annotations to ICU tokenizer helper modules

2022-07-18 09:47:57 +02:00

icu_token_analysis.py

add type annotations for ICU tokenizer

2022-07-18 09:47:57 +02:00

icu_tokenizer.py

add type annotations for ICU tokenizer

2022-07-18 09:47:57 +02:00

legacy_tokenizer.py

add type annotations for legacy tokenizer

2022-07-18 09:47:57 +02:00

place_sanitizer.py

add type hints for sanitizers

2022-07-18 09:47:57 +02:00