Sarah Hoffmann 299934fd2a reorganize and complete tests around generic token analysis

2021-10-06 17:03:37 +02:00

..

sanitizers

add tests for sanitizer tagging language

2021-10-06 12:29:25 +02:00

token_analysis

reorganize and complete tests around generic token analysis

2021-10-06 17:03:37 +02:00

__init__.py

introduce tokenizer modules

2021-04-30 11:29:57 +02:00

base.py

unify ICUNameProcessorRules and ICURuleLoader

2021-10-01 12:27:24 +02:00

factory.py

unify ICUNameProcessorRules and ICURuleLoader

2021-10-01 12:27:24 +02:00

icu_rule_loader.py

add tests for sanitizer tagging language

2021-10-06 12:29:25 +02:00

icu_token_analysis.py

use analyser provided in the 'analyzer' property

2021-10-05 14:10:32 +02:00

icu_tokenizer.py

use analyser provided in the 'analyzer' property

2021-10-05 14:10:32 +02:00

legacy_tokenizer.py

unify ICUNameProcessorRules and ICURuleLoader

2021-10-01 12:27:24 +02:00

place_sanitizer.py

introduce sanitizer step before token analysis

2021-10-01 12:27:24 +02:00