Sarah Hoffmann 7f7d2fd5b3 skip most addr: tags with suffixes ...

Only one addr: tag can be processed currently, so make
sure it is the one without suffixes to not get odd data.
addr:street is the exception because it uses a different
matching mechanism.

2021-12-06 14:55:10 +01:00

..

sanitizers

add documentation for new configuration of ICU tokenizer

2021-10-07 11:55:53 +02:00

token_analysis

reorganize and complete tests around generic token analysis

2021-10-06 17:03:37 +02:00

__init__.py

introduce tokenizer modules

2021-04-30 11:29:57 +02:00

base.py

remove unnecessary pass statements

2021-12-02 15:54:24 +01:00

factory.py

unify ICUNameProcessorRules and ICURuleLoader

2021-10-01 12:27:24 +02:00

icu_rule_loader.py

add tests for sanitizer tagging language

2021-10-06 12:29:25 +02:00

icu_token_analysis.py

use analyser provided in the 'analyzer' property

2021-10-05 14:10:32 +02:00

icu_tokenizer.py

skip most addr: tags with suffixes

2021-12-06 14:55:10 +01:00

legacy_tokenizer.py

more unit tests for tokenizers

2021-12-02 15:46:36 +01:00

place_sanitizer.py

introduce sanitizer step before token analysis

2021-10-01 12:27:24 +02:00