Sarah Hoffmann a0a7b05c9f correctly quote strings when copying in data ...

Encapsulate the copy string in a class that ensures that
copy lines are written with correct quoting.

2021-07-04 10:28:20 +02:00

..

__init__.py

introduce tokenizer modules

2021-04-30 11:29:57 +02:00

factory.py

do not hide errors when importing tokenizer

2021-05-18 16:28:21 +02:00

icu_name_processor.py

update unit tests for adapted abbreviation code

2021-07-04 10:28:20 +02:00

icu_rule_loader.py

correctly quote strings when copying in data

2021-07-04 10:28:20 +02:00

legacy_icu_tokenizer.py

correctly quote strings when copying in data

2021-07-04 10:28:20 +02:00

legacy_tokenizer.py

adapt tests for ICU tokenizer

2021-07-04 10:28:20 +02:00