Sarah Hoffmann 6c79a60e19 add documentation for new configuration of ICU tokenizer

2021-10-07 11:55:53 +02:00

..

sanitizers

add documentation for new configuration of ICU tokenizer

2021-10-07 11:55:53 +02:00

token_analysis

reorganize and complete tests around generic token analysis

2021-10-06 17:03:37 +02:00

__init__.py

introduce tokenizer modules

2021-04-30 11:29:57 +02:00

base.py

fix argument description for check_database

2021-10-07 09:49:13 +02:00

factory.py

unify ICUNameProcessorRules and ICURuleLoader

2021-10-01 12:27:24 +02:00

icu_rule_loader.py

add tests for sanitizer tagging language

2021-10-06 12:29:25 +02:00

icu_token_analysis.py

use analyser provided in the 'analyzer' property

2021-10-05 14:10:32 +02:00

icu_tokenizer.py

use analyser provided in the 'analyzer' property

2021-10-05 14:10:32 +02:00

legacy_tokenizer.py

unify ICUNameProcessorRules and ICURuleLoader

2021-10-01 12:27:24 +02:00

place_sanitizer.py

introduce sanitizer step before token analysis

2021-10-01 12:27:24 +02:00