Sarah Hoffmann b5540dc35c add migration for configurable tokenizer ...

Adds a migration that initialises a legacy tokenizer for
an existing database. The migration is not active yet as
it will need completion when more functionality is added
to the legacy tokenizer.

2021-04-30 11:29:57 +02:00

..

__init__.py

introduce tokenizer modules

2021-04-30 11:29:57 +02:00

factory.py

add migration for configurable tokenizer

2021-04-30 11:29:57 +02:00

legacy_tokenizer.py

add migration for configurable tokenizer

2021-04-30 11:29:57 +02:00