5 Commits

Author	SHA1	Message	Date
Sarah Hoffmann	6c79a60e19	add documentation for new configuration of ICU tokenizer	2021-10-07 11:55:53 +02:00
Sarah Hoffmann	118858a55e	rename legacy_icu tokenizer to icu tokenizer ... The new icu tokenizer is now no longer compatible with the old legacy tokenizer in terms of data structures. Therefore there is also no longer a need to refer to the legacy tokenizer in the name.	2021-08-17 23:11:47 +02:00
Sarah Hoffmann	34dcf02dee	fix typos in tokenizer docs	2021-07-28 11:28:49 +02:00
Sarah Hoffmann	affe1300d9	add warning about experimental nature of ICU tokenizer	2021-07-04 10:44:58 +02:00
Sarah Hoffmann	5dd24b3ef0	add documentation for ICU tokenizer configuration	2021-07-04 10:28:20 +02:00