Sarah Hoffmann 14a78f55cd more unit tests for tokenizers

2021-12-02 15:46:36 +01:00

..

sanitizers

add documentation for new configuration of ICU tokenizer

2021-10-07 11:55:53 +02:00

token_analysis

reorganize and complete tests around generic token analysis

2021-10-06 17:03:37 +02:00

__init__.py

introduce tokenizer modules

2021-04-30 11:29:57 +02:00

base.py

make word recount a tokenizer-specific function

2021-10-19 11:21:16 +02:00

factory.py

unify ICUNameProcessorRules and ICURuleLoader

2021-10-01 12:27:24 +02:00

icu_rule_loader.py

add tests for sanitizer tagging language

2021-10-06 12:29:25 +02:00

icu_token_analysis.py

use analyser provided in the 'analyzer' property

2021-10-05 14:10:32 +02:00

icu_tokenizer.py

ICU: use normalization from config in PHP

2021-10-27 11:32:44 +02:00

legacy_tokenizer.py

more unit tests for tokenizers

2021-12-02 15:46:36 +01:00

place_sanitizer.py

introduce sanitizer step before token analysis

2021-10-01 12:27:24 +02:00