Sarah Hoffmann
a3e4e8e5cd
delete unused country name tokens
2022-02-23 09:23:06 +01:00
Sarah Hoffmann
38c3ef3da0
add tests for get_string_list()
...
Renaming test file for sanitizer config because pytest requires
unique names for test files.
2022-02-07 11:22:24 +01:00
Sarah Hoffmann
610f2cc254
sanitizer: move helpers into a configuration class
2022-02-07 10:48:00 +01:00
Sarah Hoffmann
a79a3210e6
implement is-a-name option for housenumbers
2022-02-07 09:27:11 +01:00
Sarah Hoffmann
98432395c3
add migration for upcoming change to tiger tables
2022-01-27 11:48:27 +01:00
Sarah Hoffmann
83d2c440d5
add migration for new interpolation table layout
2022-01-27 11:14:55 +01:00
Sarah Hoffmann
e6d855b954
add migration for new lookup index
2022-01-27 11:14:55 +01:00
Sarah Hoffmann
c170d323d9
add tests for cleaning housenumbers
2022-01-20 23:47:20 +01:00
Sarah Hoffmann
3ce123ab69
do not clean housenumbers in reverse-only mode
2022-01-20 20:21:13 +01:00
Sarah Hoffmann
d8b7a51ab6
add actual removal of housenumber tokens
2022-01-20 20:18:15 +01:00
Sarah Hoffmann
344a2bfc1a
add new command for cleaning word tokens
...
Just pulls outdated housenumbers for the moment.
2022-01-20 20:05:15 +01:00
Sarah Hoffmann
1e5a8561c0
fix linting issues
2022-01-20 16:00:23 +01:00
Sarah Hoffmann
f3c9578bca
complete documentation for new clean-houseunubmers sanatizer
2022-01-20 15:49:32 +01:00
Sarah Hoffmann
3741afa6dc
generalize filter-kind parameter for sanatizers
...
Now behaves the same for tag_analyzer_by_language and
clean_housenumbers. Adds tests.
2022-01-20 15:42:42 +01:00
Sarah Hoffmann
4774e45218
clean_housenumbers: make kinds and delimiters configurable
...
Also adds unit tests for various options.
2022-01-20 12:07:12 +01:00
Sarah Hoffmann
206ee87188
factor out housenumber splitting into sanitizer
2022-01-19 17:27:50 +01:00
Sarah Hoffmann
3df560ea38
fix linting error
2022-01-18 11:09:21 +01:00
Sarah Hoffmann
adbaf700cd
move parsing of mutation config to setup phase
2022-01-18 11:09:21 +01:00
Sarah Hoffmann
b453b0ea95
introduce mutation variants to generic token analyser
...
Mutations are regular-expression-based replacements that are applied
after variants have been computed. They are meant to be used for
variations on character level.
Add spelling variations for German umlauts.
2022-01-18 11:09:21 +01:00
Sarah Hoffmann
0192a7af96
move variant configuration reading in separate file
2022-01-18 11:09:21 +01:00
Sarah Hoffmann
630ad38a67
refactor variant production to use generators
2022-01-18 11:09:21 +01:00
Sarah Hoffmann
c3788d765e
add consistent SPDX copyright headers
2022-01-03 16:23:58 +01:00
Sarah Hoffmann
f9b56a8581
correctly match abbreviated addr:street
...
This only works when addr:street is abbreviated and the street
name isn't. It does not work the other way around.
2021-12-08 21:58:43 +01:00
Sarah Hoffmann
7f7d2fd5b3
skip most addr: tags with suffixes
...
Only one addr: tag can be processed currently, so make
sure it is the one without suffixes to not get odd data.
addr:street is the exception because it uses a different
matching mechanism.
2021-12-06 14:55:10 +01:00
Sarah Hoffmann
44cfce1ca4
revert to using full names for street name matching
...
Using partial names turned out to not work well because there are
often similarly named streets next to each other. It also
prevents us from being able to take into account all addr:street:*
tags.
This change gets all the full term tokens for the addr:street tags
from the DB. As they are used for matching only, we can assume that
the term must already be there or there will be no match. This
avoid creating unused full name tags.
2021-12-06 11:38:38 +01:00
Sarah Hoffmann
54d35ddfe9
split cli tests by subcommand and extend coverage
2021-12-02 23:45:48 +01:00
Sarah Hoffmann
7beccb7997
remove unnecessary pass statements
2021-12-02 15:54:24 +01:00
Sarah Hoffmann
14a78f55cd
more unit tests for tokenizers
2021-12-02 15:46:36 +01:00
Sarah Hoffmann
a52ed366e4
add tests for migration
2021-12-01 20:27:40 +01:00
Sarah Hoffmann
810056349f
add migration for inclusive housenumber Tiger index
2021-11-24 12:03:20 +01:00
Sarah Hoffmann
10e979e841
only instantiate indexer once for replication
...
Also makes sure that indexer object exists everywhere were needed.
See #2518 .
2021-11-19 14:48:58 +01:00
Sarah Hoffmann
345c812e43
better error reporting when API script does not exist
...
Check if the API script exists on the expected location before
running php-cli. This way we can add a useful hint about the
project directory.
Fixes #2513 .
2021-11-10 11:58:20 +01:00
Sarah Hoffmann
d479a0585d
prepare release 4.0.0
2021-11-02 20:27:55 +01:00
Sarah Hoffmann
37eeccbf4c
ICU: use normalization from config in PHP
...
The TERM_NORMALIZATION config option is no longer applicable.
That was already documented but not yet implemented.
2021-10-27 11:32:44 +02:00
Sarah Hoffmann
53dbe58ada
do not count words when in reverse-only mode
2021-10-26 12:00:13 +02:00
Sarah Hoffmann
2c4b798f9b
further refactor setup to keep function small
2021-10-26 12:00:13 +02:00
Sarah Hoffmann
9934421442
make word count computation part of the import
...
Accurate word counts are now essential when using
the ICU tokenizer and don't hurt for the legacy one.
Adds about an hour import time.
2021-10-26 12:00:13 +02:00
Sarah Hoffmann
5c778c6d32
Merge pull request #2486 from lonvia/fix-special-phrases
...
Fix parsing of operator in special phrases
2021-10-25 21:45:08 +02:00
Sarah Hoffmann
85797acf1e
ICU: add an index over word_ids
...
Needed for keyword lookup in the details response.
2021-10-25 21:33:27 +02:00
Sarah Hoffmann
c4f5c11a4e
be case-insensitve about special phrase operator
2021-10-25 19:51:20 +02:00
Sarah Hoffmann
5a1c3dbea3
fix parsing of operator in special phrases
...
Because of unstripped input, the operators wouldn't match.
2021-10-25 19:46:30 +02:00
Sarah Hoffmann
13e7398566
allow relative paths for log files
2021-10-25 10:26:05 +02:00
Sarah Hoffmann
1098ab732f
allow relative paths for flatnode file
2021-10-22 17:32:51 +02:00
Sarah Hoffmann
507fdd4f40
switch IMPORT_STYLE to use generic file search
...
Allows relative paths wrt project directory.
2021-10-22 16:49:57 +02:00
Sarah Hoffmann
0ae8d7ac08
have ADDRESS_LEVEL_CONFIG use load_sub_configuration
...
This means that relative paths now are looked up in the
project directory.
2021-10-22 16:36:52 +02:00
Sarah Hoffmann
c77df2d1eb
replace NOMINATIM_PHRASE_CONFIG with command line option
2021-10-22 14:41:14 +02:00
Sarah Hoffmann
c1fa70639b
add new replication mode catch-up
...
This mode gets updates until the server reports no new diffs
anymore.
Also adds additional indexing, when the main indexing step left
a couple of objects to process. This happens only when the
next update is expected to be more than 40min away.
2021-10-20 22:05:15 +02:00
Sarah Hoffmann
12643c5986
run Tiger import with parallel threads per default
2021-10-19 15:00:26 +02:00
Sarah Hoffmann
ec7184c533
icu: no longer precompute terms
...
The ICU analyzer no longer drops frequent partials, so it is no
longer necessary to know the frequencies in advance.
2021-10-19 11:52:28 +02:00
Sarah Hoffmann
e8e2502e2f
make word recount a tokenizer-specific function
2021-10-19 11:21:16 +02:00
