Sarah Hoffmann 118858a55e rename legacy_icu tokenizer to icu tokenizer ...

The new icu tokenizer is now no longer compatible with the old
legacy tokenizer in terms of data structures. Therefore there
is also no longer a need to refer to the legacy tokenizer in the
name.

2021-08-17 23:11:47 +02:00

..

icu_tokenizer.php

rename legacy_icu tokenizer to icu tokenizer

2021-08-17 23:11:47 +02:00

legacy_tokenizer.php

move special hack for US states to legacy tokenizer

2021-08-17 14:28:55 +02:00