make token analyzers configurable modules

Adds a mandatory section 'analyzer' to the token-analysis entries
which define, which analyser to use. Currently there is exactly
one, generic, which implements the former ICUNameProcessor.
This commit is contained in:
Sarah Hoffmann
2021-10-04 17:34:30 +02:00
parent 52847b61a3
commit 7cfcbacfc7
7 changed files with 49 additions and 29 deletions

View File

@@ -28,7 +28,8 @@ sanitizers:
- step: split-name-list
- step: strip-brace-terms
token-analysis:
- variants:
- analyzer: generic
variants:
- !include icu-rules/variants-bg.yaml
- !include icu-rules/variants-ca.yaml
- !include icu-rules/variants-cs.yaml