mirror of
https://github.com/osm-search/Nominatim.git
synced 2026-03-13 14:24:08 +00:00
introduce external processing in indexer
Indexing is now split into three parts: first a preparation step that collects the necessary information from the database and returns it to Python. In a second step the data is transformed within Python as necessary and then returned to the database through the usual UPDATE which now not only sets the indexed_status but also other fields. The third step comprises the address computation which is still done inside the update trigger in the database. The second processing step doesn't do anything useful yet.
This commit is contained in:
@@ -187,14 +187,14 @@ class Indexer:
|
||||
|
||||
with WorkerPool(self.dsn, self.num_threads) as pool:
|
||||
while True:
|
||||
places = [p[0] for p in cur.fetchmany(batch)]
|
||||
places = [p for p in cur.fetchmany(batch)]
|
||||
if not places:
|
||||
break
|
||||
|
||||
LOG.debug("Processing places: %s", str(places))
|
||||
worker = pool.next_free_worker()
|
||||
|
||||
worker.perform(runner.sql_index_place(places))
|
||||
runner.index_places(worker, places)
|
||||
progress.add(len(places))
|
||||
|
||||
pool.finish_all()
|
||||
|
||||
Reference in New Issue
Block a user