Sarah Hoffmann f00b8dd1c3 move special hack for US states to legacy tokenizer ...

The hack for IL, AL and LA is only needed because these abbreviations
are removed by the legacy tokenizer as a stop word. There is no need
to keep the hack for future tokenizers. Move it therefore to the
token extraction function.

2021-08-17 14:28:55 +02:00

35 KiB

Raw Blame History