Sarah Hoffmann
296a66558f
move module installation to legacy tokenizer
2021-04-30 11:29:57 +02:00
Sarah Hoffmann
af968d4903
introduce tokenizer modules
...
This adds the boilerplate for selecting configurable tokenizers.
A tokenizer can be chosen at import time and will then install
itself such that it is fixed for the given database import even
when the software itself is updated.
The legacy tokenizer implements Nominatim's traditional algorithms.
2021-04-30 11:29:57 +02:00
Sarah Hoffmann
185d369404
remove support for AUX housenumber tables
...
These tables have never been actively maintained and the code is
completely untested. With the upcomming changes, it is unlikely
that the code remains usable.
This removes the aux tables and all code that references them.
2021-04-30 10:08:29 +02:00
Sarah Hoffmann
51d20b19b6
Merge pull request #2299 from lonvia/update-actions
...
Fix database check for reverse-only
2021-04-27 12:18:45 +02:00
Sarah Hoffmann
46e8c6b112
Merge pull request #2291 from AntoJvlt/special-phrases-statistics
...
Special phrases statistics
2021-04-27 11:57:05 +02:00
Sarah Hoffmann
c8fb25201a
do not check for extra housenumber index for reverse-only
...
Also adds a database check for reverse only import to the CI.
2021-04-27 10:14:26 +02:00
Sarah Hoffmann
4457bf7528
avoid Path in subprocess parameters
...
Not supported by Python 3.5.
2021-04-26 10:55:23 +02:00
AntoJvlt
abb3d56b20
Switching to log info and only send warning for invalid phrases
2021-04-25 17:57:43 +02:00
AntoJvlt
c5ecb9bae0
Implemented statistics for the import of special phrases through the SpecialPhrasesImporterStatistics class
2021-04-25 17:57:43 +02:00
AntoJvlt
1b68152fb2
reorganization of folder/file for the special phrases importer
2021-04-25 17:57:42 +02:00
Sarah Hoffmann
b951b11336
fix pylint complaints
2021-04-24 11:59:32 +02:00
Sarah Hoffmann
89c90bedb9
pylint: disable check too-few-public-methods
2021-04-24 11:39:44 +02:00
Sarah Hoffmann
9c51c133f7
indexes with includes are not available for postgresql < 11
2021-04-23 22:50:08 +02:00
Sarah Hoffmann
91d2fb6b1c
use group() for regex matches
...
Needed for compatibility with Python 3.5.
2021-04-23 22:50:08 +02:00
Sarah Hoffmann
280406c0d7
use pathlib version of open
2021-04-23 22:50:08 +02:00
Sarah Hoffmann
d5fc3b5e99
subprocess needs string argument
...
Compatibility change for Python 3.5.
2021-04-23 22:50:08 +02:00
Sarah Hoffmann
f8f8c7e534
check for existance of custom .env before opening
2021-04-23 22:50:08 +02:00
Sarah Hoffmann
3a642d50a4
use more generic ImportError to check for module
...
ModuleNotFoundError was only introduced in Python 3.6.
2021-04-23 22:50:08 +02:00
Sarah Hoffmann
9685c68e30
replace usages of fromisoformat() with strptime()
...
fromisoformat was only introduced with Python 3.7 while we
still support Python 3.5.
Fixes #2292 .
2021-04-23 22:50:08 +02:00
RhinoDevel
b7bae80616
Replace "nominatim-update" with "nominatim".
...
If I am not mistaken, the correct command to index imported data via commandline is "nominatim index".
2021-04-22 15:40:22 +02:00
Sarah Hoffmann
f7e4aa51d3
indexer: reset query counter
...
Reset the counter for queries after the asynchronous connections
have been reopened.
2021-04-21 10:33:45 +02:00
Sarah Hoffmann
50b6d7298c
factor out async connection handling into separate class
...
Also adds a test for reconnecting regularly while indexing.
2021-04-20 14:08:37 +02:00
Sarah Hoffmann
26a81654a8
indexer: make self.conn function-local
...
Also switches to our internal connect function which gives us
a cursor with a sclar() function.
2021-04-20 14:08:37 +02:00
Sarah Hoffmann
6430371d7d
make index() function private
2021-04-20 14:08:37 +02:00
Sarah Hoffmann
18705b3f18
move analyse function into indexinf function
2021-04-20 14:08:37 +02:00
Sarah Hoffmann
c6bd2bb7fb
indexer: move runner into separate file
2021-04-20 14:08:37 +02:00
Sarah Hoffmann
79d55357e8
simplify sql and website creation functions
2021-04-19 10:53:30 +02:00
Sarah Hoffmann
4fa6c0ad53
simplify constructor for SQL preprocessor
...
Use sql path from config.
2021-04-19 10:26:25 +02:00
Sarah Hoffmann
8f63f9516b
simplify interface for adding tiger data
...
Also simplifies tests using existing fixtures.
2021-04-19 10:26:25 +02:00
Sarah Hoffmann
995ba2c7c2
add library directories to config
...
Allows to reduce the number of parameters in functions that take
the config anyway.
2021-04-19 10:26:25 +02:00
AntoJvlt
b2ae715699
Only log a warning if a wrong input is detected on the wiki while importing special phrases
2021-04-17 20:19:39 +02:00
AntoJvlt
a95c748363
Fix occurence regex
2021-04-17 19:24:13 +02:00
Sarah Hoffmann
d74ae669e3
add support index when continuing import at index phase
...
Indexing scans the placex table sequentially during indexing
on the initial import. That is okay because we know that all
rows need to be processed anywhere. When continuing the import,
however, a large part might already be indexed, so that the
process spends a lot of time going through rows that are no
longer of interest. Create a supporting index for all unindexed
rows to speed up the scan. This is the same index as used later
for updates.
2021-04-17 11:07:04 +02:00
Sarah Hoffmann
da98a2102a
remove transition functions from Python
2021-04-16 18:41:14 +02:00
Sarah Hoffmann
886a01c796
port function to compute initial postcodes to Python
2021-04-16 16:11:20 +02:00
Sarah Hoffmann
76b1885595
use absolute imports in Python code
...
Relative imports are no longer officially recommended.
2021-04-16 14:20:09 +02:00
Sarah Hoffmann
c64193f839
Merge pull request #2263 from AntoJvlt/special-phrases-autoupdate
...
Implemented auto update of special phrases while importing them
2021-04-15 10:13:25 +02:00
Sarah Hoffmann
e90adfc7c3
adapt database check to new index layout
2021-04-14 17:52:59 +02:00
Sarah Hoffmann
16267dc021
add migration for new placenode geometry index
2021-04-14 17:52:59 +02:00
Darkshredder
49ee7505ed
Fix: Removed error if endstatement is wrong and improved tests
2021-04-13 15:44:12 +05:30
AntoJvlt
ae2b2cb9a5
Tests added for the auto update of special phrases during import
2021-04-12 14:35:29 +02:00
AntoJvlt
8c2f287ce4
Implemented auto update of special phrases while importing them
2021-04-12 14:30:48 +02:00
AntoJvlt
5ecae10713
Fix default languages loading
2021-04-11 22:26:31 +02:00
Sarah Hoffmann
71564fa1de
split LANGUAGES parameter before use
...
The user supplies the languages as a comma-separated list.
2021-04-09 17:48:28 +02:00
Sarah Hoffmann
492186716f
prepare 3.7.0 release
2021-04-06 21:23:29 +02:00
Sarah Hoffmann
96b0699621
add migration for transliterated housenumbers
2021-04-04 15:26:47 +02:00
Sarah Hoffmann
8d8b1d4307
use non-key index to speed up housenumber search
...
On Postgresql versions 11+ add an index to speed up the lookup
of housenumbers for terms found in search_name. This is really
just a band-aid around the query planer's interpretation of the
query.
2021-04-01 17:10:44 +02:00
Darkshredder
b7d6ae93e3
Nominatim/cli.py rebase fixes
2021-03-29 14:16:41 +05:30
Darkshredder
21b1b75b08
Rebase with master
2021-03-29 14:00:45 +05:30
Darkshredder
51e2654cd2
Added Manual page and fixed documentation
2021-03-29 13:57:13 +05:30
