Sarah Hoffmann 37eeccbf4c ICU: use normalization from config in PHP ...

The TERM_NORMALIZATION config option is no longer applicable.
That was already documented but not yet implemented.

2021-10-27 11:32:44 +02:00

..

sanitizers

add documentation for new configuration of ICU tokenizer

2021-10-07 11:55:53 +02:00

token_analysis

reorganize and complete tests around generic token analysis

2021-10-06 17:03:37 +02:00

__init__.py

introduce tokenizer modules

2021-04-30 11:29:57 +02:00

base.py

make word recount a tokenizer-specific function

2021-10-19 11:21:16 +02:00

factory.py

unify ICUNameProcessorRules and ICURuleLoader

2021-10-01 12:27:24 +02:00

icu_rule_loader.py

add tests for sanitizer tagging language

2021-10-06 12:29:25 +02:00

icu_token_analysis.py

use analyser provided in the 'analyzer' property

2021-10-05 14:10:32 +02:00

icu_tokenizer.py

ICU: use normalization from config in PHP

2021-10-27 11:32:44 +02:00

legacy_tokenizer.py

do not count words when in reverse-only mode

2021-10-26 12:00:13 +02:00

place_sanitizer.py

introduce sanitizer step before token analysis

2021-10-01 12:27:24 +02:00