3 Commits

Author	SHA1	Message	Date
Sarah Hoffmann	1f07967787	fix style issue found by flake8	2024-11-10 22:47:14 +01:00
Sarah Hoffmann	a690605a96	remove support for unindexed tokens ... This was a special feature of the legacy tokenizer who would not index very frequent tokens.	2024-09-22 10:39:10 +02:00
Sarah Hoffmann	6e89310a92	split code into submodules	2024-06-26 11:52:47 +02:00