Sarah Hoffmann a263e54b94 enable BDD tests for different tokenizers ...

The tokenizer to be used can be choosen with -DTOKENIZER.

Adapt all tests, so that they work with legacy_icu tokenizer.
Move lookup in word table to a function in the tokenizer.
Special phrases are temporarily imported from the wiki until
we have an implementation that can import from file. TIGER
tests do not work yet.

2021-05-05 10:31:51 +02:00

..

__init__.py

introduce tokenizer modules

2021-04-30 11:29:57 +02:00

factory.py

boilerplate for PHP code of tokenizer

2021-04-30 11:31:52 +02:00

legacy_icu_tokenizer.py

enable BDD tests for different tokenizers

2021-05-05 10:31:51 +02:00

legacy_tokenizer.py

enable BDD tests for different tokenizers

2021-05-05 10:31:51 +02:00