restrict partial word counting to names of reasoanble length

The partial word count does not split names to save a bit of time.
The result is that it might enounter unreasonably long names
which in truth consist of multiple words. No accurate statistics
are needed so simply restrict the count to words shorter than
75 characters.
This commit is contained in:
Sarah Hoffmann
2021-07-02 15:05:17 +02:00
parent e85f7e7aa9
commit c32551b4e0

View File

@@ -163,7 +163,9 @@ class LegacyICUTokenizer:
words = Counter()
name_proc = ICUNameProcessor(self.naming_rules)
with conn.cursor(name="words") as cur:
cur.execute("SELECT svals(name) as v, count(*) FROM place GROUP BY v")
cur.execute(""" SELECT v, count(*) FROM
(SELECT svals(name) as v FROM place)x
WHERE length(v) < 75 GROUP BY v""")
for name, cnt in cur:
terms = set()