Compare commits

..

1 Commits

Author SHA1 Message Date
Sarah Hoffmann
212b69799c adapt docs for release 2025-10-29 11:08:42 +01:00
124 changed files with 1384 additions and 2318 deletions

View File

@@ -1,12 +0,0 @@
## Summary
<!-- Describe the purpose of your pull request and, if present, link to existing issues. -->
## AI usage
<!-- Please list where and to what extent AI was used. -->
## Contributor guidelines (mandatory)
<!-- We only accept pull requests that follow our guidelines. A deliberate violation may result in a ban. -->
- [ ] I have adhered to the [coding style](https://github.com/osm-search/Nominatim/blob/master/CONTRIBUTING.md#coding-style)
- [ ] I have [tested](https://github.com/osm-search/Nominatim/blob/master/CONTRIBUTING.md#testing) the proposed changes
- [ ] I have [disclosed](https://github.com/osm-search/Nominatim/blob/master/CONTRIBUTING.md#using-ai-assisted-code-generators) above any use of AI to generate code, documentation, or the pull request description

View File

@@ -22,7 +22,7 @@ runs:
- name: Install prerequisites from apt
run: |
sudo apt-get install -y -qq python3-icu python3-datrie python3-jinja2 python3-psutil python3-dotenv python3-yaml python3-sqlalchemy python3-psycopg python3-asyncpg python3-mwparserfromhell
sudo apt-get install -y -qq python3-icu python3-datrie python3-jinja2 python3-psutil python3-dotenv python3-yaml python3-sqlalchemy python3-psycopg python3-asyncpg
shell: bash
if: inputs.dependencies == 'apt'

View File

@@ -4,7 +4,7 @@
Bugs can be reported at https://github.com/openstreetmap/Nominatim/issues.
Please always open a separate issue for each problem. In particular, do
not add your bugs to closed issues. They may look similar to you but
not add your bugs to closed issues. They may looks similar to you but
often are completely different from the maintainer's point of view.
## Workflow for Pull Requests
@@ -21,7 +21,7 @@ that you are responsible for your pull requests. You should be prepared
to get change requests because as the maintainers we have to make sure
that your contribution fits well with the rest of the code. Please make
sure that you have time to react to these comments and amend the code or
engage in a conversation. Do not expect that others will pick up your code,
engage in a conversion. Do not expect that others will pick up your code,
it will almost never happen.
Please open a separate pull request for each issue you want to address.
@@ -38,7 +38,7 @@ description or in documentation need to
1. clearly mark the AI-generated sections as such, for example, by
mentioning all use of AI in the PR description, and
2. include proof that you have run the generated code on an actual
installation of Nominatim. Adding and executing tests will not be
installation of Nominatim. Adding and excuting tests will not be
sufficient. You need to show that the code actually solves the problem
the PR claims to solve.

View File

@@ -10,13 +10,13 @@ Nominatim. Please refer to the documentation of
[Nginx](https://nginx.org/en/docs/) for background information on how
to configure it.
### Installing the required packages
!!! Note
Throughout this page, we assume your Nominatim project directory is
located in `/srv/nominatim-project`. If you have put it somewhere else,
you need to adjust the commands and configuration accordingly.
!!! warning
ASGI support in gunicorn requires at least version 25.0. If you need
to work with an older version of gunicorn, please refer to
[older Nominatim deployment documentation](https://nominatim.org/release-docs/5.2/admin/Deployment-Python/)
to learn how to run gunicorn with uvicorn.
### Installing the required packages
The Nominatim frontend is best run from its own virtual environment. If
you have already created one for the database backend during the
@@ -37,27 +37,23 @@ cd Nominatim
```
The recommended way to deploy a Python ASGI application is to run
the [gunicorn](https://gunicorn.org/) HTTP server. We use
the ASGI runner [uvicorn](https://www.uvicorn.org/)
together with [gunicorn](https://gunicorn.org/) HTTP server. We use
Falcon here as the web framework.
Add the necessary packages to your virtual environment:
``` sh
/srv/nominatim-venv/bin/pip install falcon gunicorn
/srv/nominatim-venv/bin/pip install falcon uvicorn gunicorn
```
### Setting up Nominatim as a systemd job
!!! Note
These instructions assume your Nominatim project directory is
located in `/srv/nominatim-project`. If you have put it somewhere else,
you need to adjust the commands and configuration accordingly.
Next you need to set up the service that runs the Nominatim frontend. This is
easiest done with a systemd job.
First you need to tell systemd to create a socket file to be used by
gunicorn. Create the following file `/etc/systemd/system/nominatim.socket`:
hunicorn. Create the following file `/etc/systemd/system/nominatim.socket`:
``` systemd
[Unit]
@@ -85,8 +81,10 @@ Type=simple
User=www-data
Group=www-data
WorkingDirectory=/srv/nominatim-project
ExecStart=/srv/nominatim-venv/bin/gunicorn -b unix:/run/nominatim.sock -w 4 --worker-class asgi --protocol uwsgi --worker-connections 1000 "nominatim_api.server.falcon.server:run_wsgi()"
ExecStart=/srv/nominatim-venv/bin/gunicorn -b unix:/run/nominatim.sock -w 4 -k uvicorn.workers.UvicornWorker "nominatim_api.server.falcon.server:run_wsgi()"
ExecReload=/bin/kill -s HUP $MAINPID
StandardOutput=append:/var/log/gunicorn-nominatim.log
StandardError=inherit
PrivateTmp=true
TimeoutStopSec=5
KillMode=mixed
@@ -98,10 +96,7 @@ WantedBy=multi-user.target
This sets up gunicorn with 4 workers (`-w 4` in ExecStart). Each worker runs
its own Python process using
[`NOMINATIM_API_POOL_SIZE`](../customize/Settings.md#nominatim_api_pool_size)
connections to the database to serve requests in parallel. The parameter
`--worker-connections` restricts how many requests gunicorn will queue for
each worker. This can help distribute work better when the server is under
high load.
connections to the database to serve requests in parallel.
Make the new services known to systemd and start it:
@@ -113,15 +108,13 @@ sudo systemctl enable nominatim.service
sudo systemctl start nominatim.service
```
This sets the service up so that Nominatim is automatically started
This sets the service up, so that Nominatim is automatically started
on reboot.
### Configuring nginx
To make the service available to the world, you need to proxy it through
nginx. We use the binary uwsgi protocol to speed up communication
between nginx and gunicorn. Add the following definition to the default
configuration:
nginx. Add the following definition to the default configuration:
``` nginx
upstream nominatim_service {
@@ -136,8 +129,11 @@ server {
index /search;
location / {
uwsgi_pass nominatim_service;
include uwsgi_params;
proxy_set_header Host $http_host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_redirect off;
proxy_pass http://nominatim_service;
}
}
```

View File

@@ -37,7 +37,6 @@ Furthermore the following Python libraries are required:
* [Jinja2](https://palletsprojects.com/p/jinja/)
* [PyICU](https://pypi.org/project/PyICU/)
* [PyYaml](https://pyyaml.org/) (5.1+)
* [mwparserfromhell](https://github.com/earwig/mwparserfromhell/)
These will be installed automatically when using pip installation.
@@ -111,14 +110,17 @@ Then you can install Nominatim with:
pip install nominatim-db nominatim-api
## Downloading and building Nominatim
## Downloading and building Nominatim from source
The following instructions are only relevant, if you want to build and
install Nominatim **from source**.
### Downloading the latest release
You can download the [latest release from nominatim.org](https://nominatim.org/downloads/).
The release contains all necessary files. Just unpack it.
### Downloading the latest development version
### Downloading the source for the latest development version
If you want to install latest development version from github:
@@ -132,7 +134,7 @@ The development version does not include the country grid. Download it separatel
wget -O Nominatim/data/country_osm_grid.sql.gz https://nominatim.org/data/country_grid.sql.gz
```
### Building Nominatim
### Building Nominatim from source
Nominatim is easiest to run from its own virtual environment. To create one, run:

View File

@@ -113,7 +113,6 @@ The following classifications are recognized:
| named | Consider as main tag, when the object has a primary name (see [names](#name-tags) below) |
| named_with_key | Consider as main tag, when the object has a primary name with a domain prefix. For example, if the main tag is `bridge=yes`, then it will only be added as an extra entry, if there is a tag `bridge:name[:XXX]` for the same object. If this property is set, all names that are not domain-specific are ignored. |
| fallback | Consider as main tag only when no other main tag was found. Fallback always implies `named`, i.e. fallbacks are only tried for objects with primary names. |
| postcode_area | Tag indicates a postcode area. Copy area into the table of postcodes but only when the object is a relation and has a postcode tagged. |
| delete | Completely ignore the tag in any further processing |
| extra | Move the tag to extratags and then ignore it for further processing |
| `<function>`| Advanced handling, see [below](#advanced-main-tag-handling) |

View File

@@ -229,7 +229,7 @@ _None._
| Option | Description |
|-----------------|-------------|
| locales | [Locales](../library/Result-Handling.md#locale) object for the requested language(s) |
| locales | [Locale](../library/Result-Handling.md#locale) object for the requested language(s) |
| group_hierarchy | Setting of [group_hierarchy](../api/Details.md#output-details) parameter |
| icon_base_url | (optional) URL pointing to icons as set in [NOMINATIM_MAPICON_URL](Settings.md#nominatim_mapicon_url) |

View File

@@ -79,7 +79,7 @@ the placex table. Only three columns are special:
Address interpolations are always ways in OSM, which is why there is no column
`osm_type`.
The **location_postcodes** table holds computed centroids of all postcodes that
The **location_postcode** table holds computed centroids of all postcodes that
can be found in the OSM data. The meaning of the columns is again the same
as that of the placex table.

View File

@@ -73,7 +73,7 @@ virtualenv ~/nominatim-dev-venv
types-jinja2 types-markupsafe types-psutil types-psycopg2 \
types-pygments types-pyyaml types-requests types-ujson \
types-urllib3 typing-extensions unicorn falcon starlette \
uvicorn mypy osmium aiosqlite mwparserfromhell
uvicorn mypy osmium aiosqlite
```
Now enter the virtual environment whenever you want to develop:

View File

@@ -52,15 +52,6 @@ To run the functional tests, do
pytest test/bdd
You can run a single feature file using expression matching:
pytest test/bdd -k osm2pgsql/import/entrances.feature
This even works for running single tests by adding the line number of the
scenario header like that:
pytest test/bdd -k 'osm2pgsql/import/entrances.feature and L4'
The BDD tests create databases for the tests. You can set name of the databases
through configuration variables in your `pytest.ini`:

View File

@@ -74,16 +74,15 @@ map place_addressline {
isaddress => BOOLEAN
}
map location_postcodes {
map location_postcode {
place_id => BIGINT
osm_id => BIGINT
postcode => TEXT
parent_place_id => BIGINT
rank_search => SMALLINT
rank_address => SMALLINT
indexed_status => SMALLINT
indexed_date => TIMESTAMP
geometry => GEOMETRY
centroid -> GEOMETRY
}
placex::place_id <-- search_name::place_id
@@ -95,6 +94,6 @@ search_name::nameaddress_vector --> word::word_id
place_addressline -[hidden]> location_property_osmline
search_name -[hidden]> place_addressline
location_property_osmline -[hidden]-> location_postcodes
location_property_osmline -[hidden]-> location_postcode
@enduml

View File

@@ -248,19 +248,19 @@ of the result. To do that, you first need to decide in which language the
results should be presented. As with the names in the result itself, the
places in `address_rows` contain all possible name translation for each row.
The library has a helper class `Locales` which helps extracting a name of a
The library has a helper class `Locale` which helps extracting a name of a
place in the preferred language. It takes a single parameter with a list
of language codes in the order of preference. So
``` python
locale = napi.Locales(['fr', 'en'])
locale = napi.Locale(['fr', 'en'])
```
creates a helper class that returns the name preferably in French. If that is
not possible, it tries English and eventually falls back to the default `name`
or `ref`.
The `Locales` object can be applied to a name dictionary to return the best-matching
The `Locale` object can be applied to a name dictionary to return the best-matching
name out of it:
``` python
@@ -273,7 +273,7 @@ component based on its `local_name` field. This is then utilized by the overall
which has a helper function to apply the function to all its address_row members and saves
the result in the `locale_name` field.
However, in order to set this `local_name` field in a preferred language, you must use the `Locales`
However, in order to set this `local_name` field in a preferred language, you must use the `Locale`
object which contains the function `localize_results`, which explicitly sets each `local_name field`.
``` python

View File

@@ -65,19 +65,7 @@ local table_definitions = {
{ column = 'geometry', type = 'geometry', projection = 'WGS84', not_null = true }
},
indexes = {}
},
place_postcode = {
ids = { type = 'any', id_column = 'osm_id', type_column = 'osm_type' },
columns = {
{ column = 'postcode', type = 'text', not_null = true },
{ column = 'country_code', type = 'text' },
{ column = 'centroid', type = 'point', projection = 'WGS84', not_null = true },
{ column = 'geometry', type = 'geometry', projection = 'WGS84' }
},
indexes = {
{ column = 'postcode', method = 'btree' }
}
}
}
}
local insert_row = {}
@@ -125,7 +113,6 @@ local PlaceTransform = {}
-- Special transform meanings which are interpreted elsewhere
PlaceTransform.fallback = 'fallback'
PlaceTransform.postcode_area = 'postcode_area'
PlaceTransform.delete = 'delete'
PlaceTransform.extra = 'extra'
@@ -432,25 +419,11 @@ function Place:write_place(k, v, mfunc)
return 0
end
function Place:geometry_is_valid()
function Place:write_row(k, v)
if self.geometry == nil then
self.geometry = self.geom_func(self.object)
if self.geometry == nil or self.geometry:is_null() then
self.geometry = false
return false
end
return true
end
return self.geometry ~= false
end
function Place:write_row(k, v)
if not self:geometry_is_valid() then
if self.geometry == nil or self.geometry:is_null() then
return 0
end
@@ -702,6 +675,9 @@ function module.process_tags(o)
if o.address.country ~= nil and #o.address.country ~= 2 then
o.address['country'] = nil
end
if POSTCODE_FALLBACK and fallback == nil and o.address.postcode ~= nil then
fallback = {'place', 'postcode', PlaceTransform.always}
end
if o.address.interpolation ~= nil then
o:write_place('place', 'houses', PlaceTransform.always)
@@ -709,41 +685,20 @@ function module.process_tags(o)
end
-- collect main keys
local postcode_collect = false
for k, v in pairs(o.intags) do
local ktable = MAIN_KEYS[k]
if ktable then
local ktype = ktable[v] or ktable[1]
if type(ktype) == 'function' then
o:write_place(k, v, ktype)
elseif ktype == 'postcode_area' then
postcode_collect = true
if o.object.type == 'relation'
and o.address.postcode ~= nil
and o:geometry_is_valid() then
insert_row.place_postcode{
postcode = o.address.postcode,
centroid = o.geometry:centroid(),
geometry = o.geometry
}
end
elseif ktype == 'fallback' and o.has_name then
fallback = {k, v, PlaceTransform.named}
end
end
end
if o.num_entries == 0 then
if fallback ~= nil then
o:write_place(fallback[1], fallback[2], fallback[3])
elseif POSTCODE_FALLBACK and not postcode_collect
and o.address.postcode ~= nil
and o:geometry_is_valid() then
insert_row.place_postcode{
postcode = o.address.postcode,
centroid = o.geometry:centroid()
}
end
if fallback ~= nil and o.num_entries == 0 then
o:write_place(fallback[1], fallback[2], fallback[3])
end
end

View File

@@ -117,8 +117,7 @@ module.MAIN_TAGS.all_boundaries = {
boundary = {'named',
place = 'delete',
land_area = 'delete',
protected_area = 'fallback',
postal_code = 'postcode_area'},
postal_code = 'always'},
landuse = 'fallback',
place = 'always'
}
@@ -199,7 +198,7 @@ module.MAIN_TAGS_POIS = function (group)
no = group},
landuse = {cemetery = 'always'},
leisure = {'always',
nature_reserve = 'named',
nature_reserve = 'fallback',
swimming_pool = 'named',
garden = 'named',
common = 'named',
@@ -322,6 +321,7 @@ module.NAME_TAGS = {}
module.NAME_TAGS.core = {main = {'name', 'name:*',
'int_name', 'int_name:*',
'nat_name', 'nat_name:*',
'reg_name', 'reg_name:*',
'loc_name', 'loc_name:*',
'old_name', 'old_name:*',

View File

@@ -2,7 +2,7 @@
--
-- This file is part of Nominatim. (https://nominatim.org)
--
-- Copyright (C) 2025 by the Nominatim developer community.
-- Copyright (C) 2022 by the Nominatim developer community.
-- For a full list of authors see the git log.
{% include('functions/utils.sql') %}
@@ -18,7 +18,7 @@
{% include 'functions/placex_triggers.sql' %}
{% endif %}
{% if 'location_postcodes' in db.tables %}
{% if 'location_postcode' in db.tables %}
{% include 'functions/postcode_triggers.sql' %}
{% endif %}

View File

@@ -2,7 +2,7 @@
--
-- This file is part of Nominatim. (https://nominatim.org)
--
-- Copyright (C) 2026 by the Nominatim developer community.
-- Copyright (C) 2022 by the Nominatim developer community.
-- For a full list of authors see the git log.
-- Functions for interpreting wkipedia/wikidata tags and computing importance.
@@ -166,7 +166,7 @@ BEGIN
END LOOP;
-- Nothing? Then try with the wikidata tag.
IF extratags ? 'wikidata' THEN
IF result.importance is null AND extratags ? 'wikidata' THEN
FOR match IN
{% if 'wikimedia_importance' in db.tables %}
SELECT * FROM wikimedia_importance
@@ -185,18 +185,18 @@ BEGIN
END IF;
-- Still nothing? Fall back to a default.
result.importance := 0.40001 - (rank_search::float / 75);
IF result.importance is null THEN
result.importance := 0.40001 - (rank_search::float / 75);
END IF;
{% if 'secondary_importance' in db.tables %}
FOR match IN
SELECT ST_Value(rast, centroid) as importance
FROM secondary_importance
WHERE ST_Intersects(ST_ConvexHull(rast), centroid) LIMIT 1
FROM secondary_importance
WHERE ST_Intersects(ST_ConvexHull(rast), centroid) LIMIT 1
LOOP
IF match.importance is not NULL THEN
-- Secondary importance as tie breaker with 0.0001 weight.
result.importance := result.importance + match.importance::float / 655350000;
END IF;
-- Secondary importance as tie breaker with 0.0001 weight.
result.importance := result.importance + match.importance::float / 655350000;
END LOOP;
{% endif %}

View File

@@ -2,7 +2,7 @@
--
-- This file is part of Nominatim. (https://nominatim.org)
--
-- Copyright (C) 2026 by the Nominatim developer community.
-- Copyright (C) 2022 by the Nominatim developer community.
-- For a full list of authors see the git log.
DROP TYPE IF EXISTS nearfeaturecentr CASCADE;
@@ -123,12 +123,10 @@ BEGIN
RETURN TRUE;
END IF;
IF in_rank_search <= 4 THEN
IF not in_estimate and in_country_code is not NULL THEN
INSERT INTO location_area_country (place_id, country_code, geometry)
(SELECT in_place_id, in_country_code, geom
FROM split_geometry(in_geometry) as geom);
END IF;
IF in_rank_search <= 4 and not in_estimate THEN
INSERT INTO location_area_country (place_id, country_code, geometry)
(SELECT in_place_id, in_country_code, geom
FROM split_geometry(in_geometry) as geom);
RETURN TRUE;
END IF;
@@ -214,6 +212,7 @@ DECLARE
BEGIN
{% for partition in db.partitions %}
IF in_partition = {{ partition }} THEN
DELETE FROM search_name_{{ partition }} values WHERE place_id = in_place_id;
IF in_rank_address > 0 THEN
INSERT INTO search_name_{{ partition }} (place_id, address_rank, name_vector, centroid)
values (in_place_id, in_rank_address, in_name_vector, in_geometry);
@@ -252,6 +251,7 @@ BEGIN
{% for partition in db.partitions %}
IF in_partition = {{ partition }} THEN
DELETE FROM location_road_{{ partition }} where place_id = in_place_id;
INSERT INTO location_road_{{ partition }} (partition, place_id, country_code, geometry)
values (in_partition, in_place_id, in_country_code, in_geometry);
RETURN TRUE;

View File

@@ -2,7 +2,7 @@
--
-- This file is part of Nominatim. (https://nominatim.org)
--
-- Copyright (C) 2026 by the Nominatim developer community.
-- Copyright (C) 2022 by the Nominatim developer community.
-- For a full list of authors see the git log.
CREATE OR REPLACE FUNCTION place_insert()
@@ -66,8 +66,7 @@ BEGIN
-- They get their parent from the interpolation.
UPDATE placex p SET indexed_status = 2
FROM planet_osm_ways w
WHERE w.id = NEW.osm_id and p.osm_type = 'N' and p.osm_id = any(w.nodes)
and indexed_status = 0;
WHERE w.id = NEW.osm_id and p.osm_type = 'N' and p.osm_id = any(w.nodes);
-- If there is already an entry in place, just update that, if necessary.
IF existing.osm_type is not null THEN
@@ -90,6 +89,35 @@ BEGIN
RETURN NEW;
END IF;
-- ---- Postcode points.
IF NEW.class = 'place' AND NEW.type = 'postcode' THEN
-- Pure postcodes are never queried from placex so we don't add them.
-- location_postcodes is filled from the place table directly.
-- Remove any old placex entry.
DELETE FROM placex WHERE osm_type = NEW.osm_type and osm_id = NEW.osm_id;
IF existing.osm_type IS NOT NULL THEN
IF coalesce(existing.address, ''::hstore) != coalesce(NEW.address, ''::hstore)
OR existing.geometry::text != NEW.geometry::text
THEN
UPDATE place
SET name = NEW.name,
address = NEW.address,
extratags = NEW.extratags,
admin_level = NEW.admin_level,
geometry = NEW.geometry
WHERE osm_type = NEW.osm_type and osm_id = NEW.osm_id
and class = NEW.class and type = NEW.type;
END IF;
RETURN NULL;
END IF;
RETURN NEW;
END IF;
-- ---- All other place types.
-- When an area is changed from large to small: log and discard change
@@ -241,6 +269,17 @@ BEGIN
WHERE osm_type = NEW.osm_type and osm_id = NEW.osm_id
and class = NEW.class and type = NEW.type;
-- Postcode areas are only kept, when there is an actual postcode assigned.
IF NEW.class = 'boundary' AND NEW.type = 'postal_code' THEN
IF NEW.address is NULL OR NOT NEW.address ? 'postcode' THEN
-- postcode was deleted, no longer retain in placex
DELETE FROM placex where place_id = existingplacex.place_id;
RETURN NULL;
END IF;
NEW.name := hstore('ref', NEW.address->'postcode');
END IF;
-- Boundaries must be areas.
IF NEW.class in ('boundary')
AND ST_GeometryType(NEW.geometry) not in ('ST_Polygon','ST_MultiPolygon')

View File

@@ -2,7 +2,7 @@
--
-- This file is part of Nominatim. (https://nominatim.org)
--
-- Copyright (C) 2026 by the Nominatim developer community.
-- Copyright (C) 2025 by the Nominatim developer community.
-- For a full list of authors see the git log.
-- Trigger functions for the placex table.
@@ -304,6 +304,7 @@ DECLARE
BEGIN
IF bnd.rank_search >= 26 or bnd.rank_address = 0
or ST_GeometryType(bnd.geometry) NOT IN ('ST_Polygon','ST_MultiPolygon')
or bnd.type IN ('postcode', 'postal_code')
THEN
RETURN NULL;
END IF;
@@ -340,25 +341,10 @@ BEGIN
END IF;
END IF;
IF bnd.extratags ? 'wikidata' THEN
FOR linked_placex IN
SELECT * FROM placex
WHERE placex.class = 'place' AND placex.osm_type = 'N'
AND placex.extratags ? 'wikidata' -- needed to select right index
AND placex.extratags->'wikidata' = bnd.extratags->'wikidata'
AND (placex.linked_place_id is null or placex.linked_place_id = bnd.place_id)
AND placex.rank_search < 26
AND _st_covers(bnd.geometry, placex.geometry)
ORDER BY lower(name->'name') = bnd_name desc
LOOP
{% if debug %}RAISE WARNING 'Found wikidata-matching place node %', linked_placex.osm_id;{% endif %}
RETURN linked_placex;
END LOOP;
END IF;
-- If extratags has a place tag, look for linked nodes by their place type.
-- Area and node still have to have the same name.
IF bnd.extratags ? 'place' and bnd_name is not null
IF bnd.extratags ? 'place' and bnd.extratags->'place' != 'postcode'
and bnd_name is not null
THEN
FOR linked_placex IN
SELECT * FROM placex
@@ -375,6 +361,22 @@ BEGIN
END LOOP;
END IF;
IF bnd.extratags ? 'wikidata' THEN
FOR linked_placex IN
SELECT * FROM placex
WHERE placex.class = 'place' AND placex.osm_type = 'N'
AND placex.extratags ? 'wikidata' -- needed to select right index
AND placex.extratags->'wikidata' = bnd.extratags->'wikidata'
AND (placex.linked_place_id is null or placex.linked_place_id = bnd.place_id)
AND placex.rank_search < 26
AND _st_covers(bnd.geometry, placex.geometry)
ORDER BY lower(name->'name') = bnd_name desc
LOOP
{% if debug %}RAISE WARNING 'Found wikidata-matching place node %', linked_placex.osm_id;{% endif %}
RETURN linked_placex;
END LOOP;
END IF;
-- Name searches can be done for ways as well as relations
IF bnd_name is not null THEN
{% if debug %}RAISE WARNING 'Looking for nodes with matching names';{% endif %}
@@ -391,6 +393,7 @@ BEGIN
AND placex.class = 'place'
AND (placex.linked_place_id is null or placex.linked_place_id = bnd.place_id)
AND placex.rank_search < 26 -- needed to select the right index
AND placex.type != 'postcode'
AND ST_Covers(bnd.geometry, placex.geometry)
LOOP
{% if debug %}RAISE WARNING 'Found matching place node %', linked_placex.osm_id;{% endif %}
@@ -465,7 +468,7 @@ BEGIN
END IF;
END LOOP;
name_vector := COALESCE(token_get_name_search_tokens(token_info), '{}'::INTEGER[]);
name_vector := token_get_name_search_tokens(token_info);
-- Check if the parent covers all address terms.
-- If not, create a search name entry with the house number as the name.
@@ -694,7 +697,17 @@ BEGIN
ELSE
is_area := ST_GeometryType(NEW.geometry) IN ('ST_Polygon','ST_MultiPolygon');
IF NEW.class = 'highway' AND is_area AND NEW.name is null
IF NEW.class in ('place','boundary')
AND NEW.type in ('postcode','postal_code')
THEN
IF NEW.address IS NULL OR NOT NEW.address ? 'postcode' THEN
-- most likely just a part of a multipolygon postcode boundary, throw it away
RETURN NULL;
END IF;
NEW.name := hstore('ref', NEW.address->'postcode');
ELSEIF NEW.class = 'highway' AND is_area AND NEW.name is null
AND NEW.extratags ? 'area' AND NEW.extratags->'area' = 'yes'
THEN
RETURN NULL;
@@ -840,15 +853,13 @@ BEGIN
NEW.indexed_date = now();
IF OLD.indexed_status > 1 THEN
{% if 'search_name' in db.tables %}
DELETE from search_name WHERE place_id = NEW.place_id;
{% endif %}
result := deleteSearchName(NEW.partition, NEW.place_id);
DELETE FROM place_addressline WHERE place_id = NEW.place_id;
result := deleteRoad(NEW.partition, NEW.place_id);
result := deleteLocationArea(NEW.partition, NEW.place_id, NEW.rank_search);
END IF;
{% if 'search_name' in db.tables %}
DELETE from search_name WHERE place_id = NEW.place_id;
{% endif %}
result := deleteSearchName(NEW.partition, NEW.place_id);
DELETE FROM place_addressline WHERE place_id = NEW.place_id;
result := deleteRoad(NEW.partition, NEW.place_id);
result := deleteLocationArea(NEW.partition, NEW.place_id, NEW.rank_search);
NEW.extratags := NEW.extratags - 'linked_place'::TEXT;
IF NEW.extratags = ''::hstore THEN
@@ -861,12 +872,19 @@ BEGIN
NEW.linked_place_id := OLD.linked_place_id;
-- Remove linkage, if we have computed a different new linkee.
IF OLD.indexed_status > 1 THEN
UPDATE placex
SET linked_place_id = null,
indexed_status = CASE WHEN indexed_status = 0 THEN 2 ELSE indexed_status END
WHERE linked_place_id = NEW.place_id
and (linked_place is null or place_id != linked_place);
UPDATE placex SET linked_place_id = null, indexed_status = 2
WHERE linked_place_id = NEW.place_id
and (linked_place is null or linked_place_id != linked_place);
-- update not necessary for osmline, cause linked_place_id does not exist
-- Postcodes are just here to compute the centroids. They are not searchable
-- unless they are a boundary=postal_code.
-- There was an error in the style so that boundary=postal_code used to be
-- imported as place=postcode. That's why relations are allowed to pass here.
-- This can go away in a couple of versions.
IF NEW.class = 'place' and NEW.type = 'postcode' and NEW.osm_type != 'R' THEN
NEW.token_info := null;
RETURN NEW;
END IF;
-- Compute a preliminary centroid.
@@ -1037,9 +1055,7 @@ BEGIN
LOOP
UPDATE placex SET linked_place_id = NEW.place_id WHERE place_id = linked_node_id;
{% if 'search_name' in db.tables %}
IF OLD.indexed_status > 1 THEN
DELETE FROM search_name WHERE place_id = linked_node_id;
END IF;
DELETE FROM search_name WHERE place_id = linked_node_id;
{% endif %}
END LOOP;
END IF;
@@ -1188,6 +1204,11 @@ BEGIN
-- reset the address rank if necessary.
UPDATE placex set linked_place_id = NEW.place_id, indexed_status = 2
WHERE place_id = location.place_id;
-- ensure that those places are not found anymore
{% if 'search_name' in db.tables %}
DELETE FROM search_name WHERE place_id = location.place_id;
{% endif %}
PERFORM deleteLocationArea(NEW.partition, location.place_id, NEW.rank_search);
SELECT wikipedia, importance
FROM compute_importance(location.extratags, NEW.country_code,
@@ -1198,7 +1219,7 @@ BEGIN
IF linked_importance is not null AND
(NEW.importance is null or NEW.importance < linked_importance)
THEN
NEW.importance := linked_importance;
NEW.importance = linked_importance;
END IF;
ELSE
-- No linked place? As a last resort check if the boundary is tagged with
@@ -1240,7 +1261,7 @@ BEGIN
LIMIT 1
LOOP
IF location.osm_id = NEW.osm_id THEN
{% if debug %}RAISE WARNING 'Updating names for country ''%'' with: %', NEW.country_code, NEW.name;{% endif %}
{% if debug %}RAISE WARNING 'Updating names for country '%' with: %', NEW.country_code, NEW.name;{% endif %}
UPDATE country_name SET derived_name = NEW.name WHERE country_code = NEW.country_code;
END IF;
END LOOP;
@@ -1265,6 +1286,8 @@ BEGIN
END IF;
ELSEIF NEW.rank_address > 25 THEN
max_rank := 25;
ELSEIF NEW.class in ('place','boundary') and NEW.type in ('postcode','postal_code') THEN
max_rank := NEW.rank_search;
ELSE
max_rank := NEW.rank_address;
END IF;
@@ -1279,10 +1302,10 @@ BEGIN
NEW.postcode := coalesce(token_get_postcode(NEW.token_info), NEW.postcode);
-- if we have a name add this to the name search table
name_vector := token_get_name_search_tokens(NEW.token_info);
IF array_length(name_vector, 1) is not NULL THEN
IF NEW.name IS NOT NULL THEN
-- Initialise the name vector using our name
NEW.name := add_default_place_name(NEW.country_code, NEW.name);
name_vector := token_get_name_search_tokens(NEW.token_info);
IF NEW.rank_search <= 25 and NEW.rank_address > 0 THEN
result := add_location(NEW.place_id, NEW.country_code, NEW.partition,
@@ -1342,10 +1365,10 @@ BEGIN
-- RAISE WARNING 'placex_delete % %',OLD.osm_type,OLD.osm_id;
IF OLD.linked_place_id is null THEN
UPDATE placex
SET linked_place_id = NULL,
indexed_status = CASE WHEN indexed_status = 0 THEN 2 ELSE indexed_status END
WHERE linked_place_id = OLD.place_id;
update placex set linked_place_id = null, indexed_status = 2 where linked_place_id = OLD.place_id and indexed_status = 0;
{% if debug %}RAISE WARNING 'placex_delete:01 % %',OLD.osm_type,OLD.osm_id;{% endif %}
update placex set linked_place_id = null where linked_place_id = OLD.place_id;
{% if debug %}RAISE WARNING 'placex_delete:02 % %',OLD.osm_type,OLD.osm_id;{% endif %}
ELSE
update placex set indexed_status = 2 where place_id = OLD.linked_place_id and indexed_status = 0;
END IF;
@@ -1369,7 +1392,6 @@ BEGIN
-- reparenting also for OSM Interpolation Lines (and for Tiger?)
update location_property_osmline set indexed_status = 2 where indexed_status = 0 and parent_place_id = OLD.place_id;
UPDATE location_postcodes SET indexed_status = 2 WHERE parent_place_id = OLD.place_id;
END IF;
{% if debug %}RAISE WARNING 'placex_delete:08 % %',OLD.osm_type,OLD.osm_id;{% endif %}
@@ -1401,6 +1423,9 @@ BEGIN
END IF;
{% if debug %}RAISE WARNING 'placex_delete:12 % %',OLD.osm_type,OLD.osm_id;{% endif %}
UPDATE location_postcode SET indexed_status = 2 WHERE parent_place_id = OLD.place_id;
RETURN OLD;
END;

View File

@@ -2,10 +2,10 @@
--
-- This file is part of Nominatim. (https://nominatim.org)
--
-- Copyright (C) 2025 by the Nominatim developer community.
-- Copyright (C) 2022 by the Nominatim developer community.
-- For a full list of authors see the git log.
-- Trigger functions for location_postcodes table.
-- Trigger functions for location_postcode table.
-- Trigger for updates of location_postcode
@@ -13,7 +13,7 @@
-- Computes the parent object the postcode most likely refers to.
-- This will be the place that determines the address displayed when
-- searching for this postcode.
CREATE OR REPLACE FUNCTION postcodes_update()
CREATE OR REPLACE FUNCTION postcode_update()
RETURNS TRIGGER
AS $$
DECLARE
@@ -28,10 +28,13 @@ BEGIN
partition := get_partition(NEW.country_code);
SELECT * FROM get_postcode_rank(NEW.country_code, NEW.postcode)
INTO NEW.rank_search, NEW.rank_address;
NEW.parent_place_id = 0;
FOR location IN
SELECT place_id
FROM getNearFeatures(partition, NEW.centroid, NEW.centroid, NEW.rank_search)
FROM getNearFeatures(partition, NEW.geometry, NEW.geometry, NEW.rank_search)
WHERE NOT isguess ORDER BY rank_address DESC, distance asc LIMIT 1
LOOP
NEW.parent_place_id = location.place_id;
@@ -42,89 +45,3 @@ END;
$$
LANGUAGE plpgsql;
CREATE OR REPLACE FUNCTION postcodes_delete()
RETURNS TRIGGER
AS $$
BEGIN
{% if not disable_diff_updates %}
UPDATE placex p SET indexed_status = 2
WHERE p.postcode = OLD.postcode AND ST_Intersects(OLD.geometry, p.geometry)
AND indexed_status = 0;
{% endif %}
RETURN OLD;
END;
$$
LANGUAGE plpgsql;
CREATE OR REPLACE FUNCTION postcodes_insert()
RETURNS TRIGGER
AS $$
DECLARE
existing RECORD;
BEGIN
IF NEW.osm_id is not NULL THEN
-- postcode area, remove existing from same OSM object
SELECT * INTO existing FROM location_postcodes p
WHERE p.osm_id = NEW.osm_id;
IF existing.place_id is not NULL THEN
IF existing.postcode != NEW.postcode or existing.country_code != NEW.country_code THEN
DELETE FROM location_postcodes p WHERE p.osm_id = NEW.osm_id;
existing := NULL;
END IF;
END IF;
END IF;
IF existing is NULL THEN
SELECT * INTO existing FROM location_postcodes p
WHERE p.country_code = NEW.country_code AND p.postcode = NEW.postcode;
IF existing.postcode is NULL THEN
{% if not disable_diff_updates %}
UPDATE placex p SET indexed_status = 2
WHERE ST_Intersects(NEW.geometry, p.geometry)
AND indexed_status = 0
AND p.rank_address >= 22 AND not p.address ? 'postcode';
{% endif %}
-- new entry, just insert
NEW.indexed_status := 1;
NEW.place_id := nextval('seq_place');
RETURN NEW;
END IF;
END IF;
-- update: only when there are changes
IF coalesce(NEW.osm_id, -1) != coalesce(existing.osm_id, -1)
OR (NEW.osm_id is not null AND NEW.geometry::text != existing.geometry::text)
OR (NEW.osm_id is null
AND (abs(ST_X(existing.centroid) - ST_X(NEW.centroid)) > 0.0000001
OR abs(ST_Y(existing.centroid) - ST_Y(NEW.centroid)) > 0.0000001))
THEN
{% if not disable_diff_updates %}
UPDATE placex p SET indexed_status = 2
WHERE ST_Intersects(ST_Difference(NEW.geometry, existing.geometry), p.geometry)
AND indexed_status = 0
AND p.rank_address >= 22 AND not p.address ? 'postcode';
UPDATE placex p SET indexed_status = 2
WHERE ST_Intersects(ST_Difference(existing.geometry, NEW.geometry), p.geometry)
AND indexed_status = 0
AND p.postcode = OLD.postcode;
{% endif %}
UPDATE location_postcodes p
SET osm_id = NEW.osm_id,
indexed_status = 2,
centroid = NEW.centroid,
geometry = NEW.geometry
WHERE p.country_code = NEW.country_code AND p.postcode = NEW.postcode;
END IF;
RETURN NULL;
END;
$$
LANGUAGE plpgsql;

View File

@@ -2,7 +2,7 @@
--
-- This file is part of Nominatim. (https://nominatim.org)
--
-- Copyright (C) 2025 by the Nominatim developer community.
-- Copyright (C) 2022 by the Nominatim developer community.
-- For a full list of authors see the git log.
-- Functions related to search and address ranks
@@ -114,6 +114,66 @@ $$
LANGUAGE plpgsql IMMUTABLE PARALLEL SAFE;
-- Guess a ranking for postcodes from country and postcode format.
CREATE OR REPLACE FUNCTION get_postcode_rank(country_code VARCHAR(2), postcode TEXT,
OUT rank_search SMALLINT,
OUT rank_address SMALLINT)
AS $$
DECLARE
part TEXT;
BEGIN
rank_search := 30;
rank_address := 30;
postcode := upper(postcode);
IF country_code = 'gb' THEN
IF postcode ~ '^([A-Z][A-Z]?[0-9][0-9A-Z]? [0-9][A-Z][A-Z])$' THEN
rank_search := 25;
rank_address := 5;
ELSEIF postcode ~ '^([A-Z][A-Z]?[0-9][0-9A-Z]? [0-9])$' THEN
rank_search := 23;
rank_address := 5;
ELSEIF postcode ~ '^([A-Z][A-Z]?[0-9][0-9A-Z])$' THEN
rank_search := 21;
rank_address := 5;
END IF;
ELSEIF country_code = 'sg' THEN
IF postcode ~ '^([0-9]{6})$' THEN
rank_search := 25;
rank_address := 11;
END IF;
ELSEIF country_code = 'de' THEN
IF postcode ~ '^([0-9]{5})$' THEN
rank_search := 21;
rank_address := 11;
END IF;
ELSE
-- Guess at the postcode format and coverage (!)
IF postcode ~ '^[A-Z0-9]{1,5}$' THEN -- Probably too short to be very local
rank_search := 21;
rank_address := 11;
ELSE
-- Does it look splitable into and area and local code?
part := substring(postcode from '^([- :A-Z0-9]+)([- :][A-Z0-9]+)$');
IF part IS NOT NULL THEN
rank_search := 25;
rank_address := 11;
ELSEIF postcode ~ '^[- :A-Z0-9]{6,}$' THEN
rank_search := 21;
rank_address := 11;
END IF;
END IF;
END IF;
END;
$$
LANGUAGE plpgsql IMMUTABLE PARALLEL SAFE;
-- Get standard search and address rank for an object.
--
-- \param country Two-letter country code where the object is in.
@@ -138,7 +198,12 @@ AS $$
DECLARE
classtype TEXT;
BEGIN
IF extended_type = 'N' AND place_class = 'highway' THEN
IF place_class in ('place','boundary')
and place_type in ('postcode','postal_code')
THEN
SELECT * INTO search_rank, address_rank
FROM get_postcode_rank(country, postcode);
ELSEIF extended_type = 'N' AND place_class = 'highway' THEN
search_rank = 30;
address_rank = 30;
ELSEIF place_class = 'landuse' AND extended_type != 'A' THEN

View File

@@ -2,7 +2,7 @@
--
-- This file is part of Nominatim. (https://nominatim.org)
--
-- Copyright (C) 2026 by the Nominatim developer community.
-- Copyright (C) 2025 by the Nominatim developer community.
-- For a full list of authors see the git log.
-- Assorted helper functions for the triggers.
@@ -46,13 +46,13 @@ DECLARE
r INTEGER[];
BEGIN
IF array_upper(a, 1) IS NULL THEN
RETURN COALESCE(b, '{}'::INTEGER[]);
RETURN b;
END IF;
IF array_upper(b, 1) IS NULL THEN
RETURN COALESCE(a, '{}'::INTEGER[]);
RETURN a;
END IF;
r := a;
FOR i IN 1..array_upper(b, 1) LOOP
FOR i IN 1..array_upper(b, 1) LOOP
IF NOT (ARRAY[b[i]] <@ r) THEN
r := r || b[i];
END IF;
@@ -139,46 +139,37 @@ $$
LANGUAGE plpgsql IMMUTABLE PARALLEL SAFE;
-- Find the best-matching postcode for the given geometry
-- Find the nearest artificial postcode for the given geometry.
-- TODO For areas there should not be more than two inside the geometry.
CREATE OR REPLACE FUNCTION get_nearest_postcode(country VARCHAR(2), geom GEOMETRY)
RETURNS TEXT
AS $$
DECLARE
outcode TEXT;
cnt INTEGER;
location RECORD;
BEGIN
-- If the geometry is an area then only one postcode must be within
-- that area, otherwise consider the area as not having a postcode.
IF ST_GeometryType(geom) in ('ST_Polygon','ST_MultiPolygon') THEN
SELECT min(postcode), count(*) FROM
(SELECT postcode FROM location_postcodes
WHERE geom && location_postcodes.geometry -- want to use the index
AND ST_Contains(geom, location_postcodes.centroid)
AND country_code = country
LIMIT 2) sub
INTO outcode, cnt;
SELECT min(postcode), count(*) FROM
(SELECT postcode FROM location_postcode
WHERE ST_Contains(geom, location_postcode.geometry) LIMIT 2) sub
INTO outcode, cnt;
IF cnt = 1 THEN
RETURN outcode;
ELSE
RETURN null;
END IF;
RETURN null;
END IF;
-- Otherwise: be fully within the coverage area of a postcode
FOR location IN
SELECT postcode
FROM location_postcodes p
WHERE ST_Covers(p.geometry, geom)
AND p.country_code = country
ORDER BY osm_id is null, ST_Distance(p.centroid, geom)
LIMIT 1
LOOP
RETURN location.postcode;
END LOOP;
SELECT postcode FROM location_postcode
WHERE ST_DWithin(geom, location_postcode.geometry, 0.05)
AND location_postcode.country_code = country
ORDER BY ST_Distance(geom, location_postcode.geometry) LIMIT 1
INTO outcode;
RETURN NULL;
RETURN outcode;
END;
$$
LANGUAGE plpgsql STABLE PARALLEL SAFE;
@@ -323,17 +314,6 @@ END;
$$
LANGUAGE plpgsql;
-- Return the bounding box of the geometry buffered by the given number
-- of meters.
CREATE OR REPLACE FUNCTION expand_by_meters(geom GEOMETRY, meters FLOAT)
RETURNS GEOMETRY
AS $$
SELECT ST_Envelope(ST_Buffer(geom::geography, meters, 1)::geometry)
$$
LANGUAGE sql IMMUTABLE PARALLEL SAFE;
-- Create a bounding box with an extent computed from the radius (in meters)
-- which in turn is derived from the given search rank.
CREATE OR REPLACE FUNCTION place_node_fuzzy_area(geom GEOMETRY, rank_search INTEGER)
@@ -352,7 +332,9 @@ BEGIN
radius := 1000;
END IF;
RETURN expand_by_meters(geom, radius);
RETURN ST_Envelope(ST_Collect(
ST_Project(geom::geography, radius, 0.785398)::geometry,
ST_Project(geom::geography, radius, 3.9269908)::geometry));
END;
$$
LANGUAGE plpgsql IMMUTABLE PARALLEL SAFE;
@@ -368,6 +350,8 @@ CREATE OR REPLACE FUNCTION add_location(place_id BIGINT, country_code varchar(2)
DECLARE
postcode TEXT;
BEGIN
PERFORM deleteLocationArea(partition, place_id, rank_search);
-- add postcode only if it contains a single entry, i.e. ignore postcode lists
postcode := NULL;
IF in_postcode is not null AND in_postcode not similar to '%(,|;)%' THEN

View File

@@ -1,47 +0,0 @@
-- SPDX-License-Identifier: GPL-2.0-only
--
-- This file is part of Nominatim. (https://nominatim.org)
--
-- Copyright (C) 2026 by the Nominatim developer community.
-- For a full list of authors see the git log.
--
-- Grant read-only access to the web user for all Nominatim tables.
-- Core tables
GRANT SELECT ON import_status TO "{{config.DATABASE_WEBUSER}}";
GRANT SELECT ON country_name TO "{{config.DATABASE_WEBUSER}}";
GRANT SELECT ON nominatim_properties TO "{{config.DATABASE_WEBUSER}}";
-- Location tables
GRANT SELECT ON location_property_tiger TO "{{config.DATABASE_WEBUSER}}";
GRANT SELECT ON location_property_osmline TO "{{config.DATABASE_WEBUSER}}";
GRANT SELECT ON location_postcodes TO "{{config.DATABASE_WEBUSER}}";
-- Search tables
{% if not db.reverse_only %}
GRANT SELECT ON search_name TO "{{config.DATABASE_WEBUSER}}";
{% endif %}
-- Main place tables
GRANT SELECT ON placex TO "{{config.DATABASE_WEBUSER}}";
GRANT SELECT ON place_addressline TO "{{config.DATABASE_WEBUSER}}";
GRANT SELECT ON placex_entrance TO "{{config.DATABASE_WEBUSER}}";
-- Error/delete tracking tables
GRANT SELECT ON import_polygon_error TO "{{config.DATABASE_WEBUSER}}";
GRANT SELECT ON import_polygon_delete TO "{{config.DATABASE_WEBUSER}}";
-- Country grid
GRANT SELECT ON country_osm_grid TO "{{config.DATABASE_WEBUSER}}";
-- Tokenizer tables (word table)
{% if 'word' in db.tables %}
GRANT SELECT ON word TO "{{config.DATABASE_WEBUSER}}";
{% endif %}
-- Special phrase tables
{% for table in db.tables %}
{% if table.startswith('place_classtype_') %}
GRANT SELECT ON {{ table }} TO "{{config.DATABASE_WEBUSER}}";
{% endif %}
{% endfor %}

View File

@@ -2,7 +2,7 @@
--
-- This file is part of Nominatim. (https://nominatim.org)
--
-- Copyright (C) 2025 by the Nominatim developer community.
-- Copyright (C) 2022 by the Nominatim developer community.
-- For a full list of authors see the git log.
-- Indices used only during search and update.
@@ -21,25 +21,30 @@ CREATE INDEX IF NOT EXISTS idx_placex_parent_place_id
ON placex USING BTREE (parent_place_id) {{db.tablespace.search_index}}
WHERE parent_place_id IS NOT NULL;
---
-- Used to find postcode areas after a search in location_postcode.
CREATE INDEX IF NOT EXISTS idx_placex_postcode_areas
ON placex USING BTREE (country_code, postcode) {{db.tablespace.search_index}}
WHERE osm_type = 'R' AND class = 'boundary' AND type = 'postal_code';
---
CREATE INDEX IF NOT EXISTS idx_placex_geometry ON placex
USING GIST (geometry) {{db.tablespace.search_index}};
---
-- Index is needed during import but can be dropped as soon as a full
-- geometry index is in place. The partial index is almost as big as the full
-- index.
---
DROP INDEX IF EXISTS idx_placex_geometry_lower_rank_ways;
---
CREATE INDEX IF NOT EXISTS idx_placex_geometry_reverse_lookupPolygon
ON placex USING gist (geometry) {{db.tablespace.search_index}}
WHERE St_GeometryType(geometry) in ('ST_Polygon', 'ST_MultiPolygon')
AND rank_address between 4 and 25
AND rank_address between 4 and 25 AND type != 'postcode'
AND name is not null AND indexed_status = 0 AND linked_place_id is null;
---
-- used in reverse large area lookup
CREATE INDEX IF NOT EXISTS idx_placex_geometry_reverse_lookupPlaceNode
ON placex USING gist (ST_Buffer(geometry, reverse_place_diameter(rank_search)))
{{db.tablespace.search_index}}
WHERE rank_address between 4 and 25
WHERE rank_address between 4 and 25 AND type != 'postcode'
AND name is not null AND linked_place_id is null AND osm_type = 'N';
---
CREATE INDEX IF NOT EXISTS idx_osmline_parent_place_id
@@ -48,6 +53,9 @@ CREATE INDEX IF NOT EXISTS idx_osmline_parent_place_id
---
CREATE INDEX IF NOT EXISTS idx_osmline_parent_osm_id
ON location_property_osmline USING BTREE (osm_id) {{db.tablespace.search_index}};
---
CREATE INDEX IF NOT EXISTS idx_postcode_postcode
ON location_postcode USING BTREE (postcode) {{db.tablespace.search_index}};
{% if drop %}
---
@@ -74,8 +82,8 @@ CREATE INDEX IF NOT EXISTS idx_osmline_parent_osm_id
deferred BOOLEAN
);
---
CREATE INDEX IF NOT EXISTS idx_location_postcodes_parent_place_id
ON location_postcodes USING BTREE (parent_place_id) {{db.tablespace.address_index}};
CREATE INDEX IF NOT EXISTS idx_location_postcode_parent_place_id
ON location_postcode USING BTREE (parent_place_id) {{db.tablespace.address_index}};
{% endif %}
-- Indices only needed for search.

View File

@@ -2,48 +2,36 @@
--
-- This file is part of Nominatim. (https://nominatim.org)
--
-- Copyright (C) 2026 by the Nominatim developer community.
-- Copyright (C) 2022 by the Nominatim developer community.
-- For a full list of authors see the git log.
drop table IF EXISTS search_name_blank CASCADE;
CREATE TABLE search_name_blank (
place_id BIGINT NOT NULL,
address_rank smallint NOT NULL,
name_vector integer[] NOT NULL,
centroid GEOMETRY(Geometry, 4326) NOT NULL
place_id BIGINT,
address_rank smallint,
name_vector integer[],
centroid GEOMETRY(Geometry, 4326)
);
{% for partition in db.partitions %}
CREATE TABLE location_area_large_{{ partition }} () INHERITS (location_area_large) {{db.tablespace.address_data}};
CREATE INDEX idx_location_area_large_{{ partition }}_place_id
ON location_area_large_{{ partition }}
USING BTREE (place_id) {{db.tablespace.address_index}};
CREATE INDEX idx_location_area_large_{{ partition }}_geometry
ON location_area_large_{{ partition }}
USING GIST (geometry) {{db.tablespace.address_index}};
CREATE INDEX idx_location_area_large_{{ partition }}_place_id ON location_area_large_{{ partition }} USING BTREE (place_id) {{db.tablespace.address_index}};
CREATE INDEX idx_location_area_large_{{ partition }}_geometry ON location_area_large_{{ partition }} USING GIST (geometry) {{db.tablespace.address_index}};
CREATE TABLE search_name_{{ partition }} () INHERITS (search_name_blank) {{db.tablespace.address_data}};
CREATE UNIQUE INDEX idx_search_name_{{ partition }}_place_id
ON search_name_{{ partition }}
USING BTREE (place_id) {{db.tablespace.address_index}};
CREATE INDEX idx_search_name_{{ partition }}_centroid_street
ON search_name_{{ partition }} USING GIST (centroid) {{db.tablespace.address_index}}
WHERE address_rank between 26 and 27;
CREATE INDEX idx_search_name_{{ partition }}_centroid_place
ON search_name_{{ partition }} USING GIST (centroid) {{db.tablespace.address_index}}
WHERE address_rank between 2 and 25;
CREATE INDEX idx_search_name_{{ partition }}_place_id ON search_name_{{ partition }} USING BTREE (place_id) {{db.tablespace.address_index}};
CREATE INDEX idx_search_name_{{ partition }}_centroid_street ON search_name_{{ partition }} USING GIST (centroid) {{db.tablespace.address_index}} where address_rank between 26 and 27;
CREATE INDEX idx_search_name_{{ partition }}_centroid_place ON search_name_{{ partition }} USING GIST (centroid) {{db.tablespace.address_index}} where address_rank between 2 and 25;
DROP TABLE IF EXISTS location_road_{{ partition }};
CREATE TABLE location_road_{{ partition }} (
place_id BIGINT NOT NULL,
partition SMALLINT NOT NULL,
place_id BIGINT,
partition SMALLINT,
country_code VARCHAR(2),
geometry GEOMETRY(Geometry, 4326) NOT NULL
geometry GEOMETRY(Geometry, 4326)
) {{db.tablespace.address_data}};
CREATE INDEX idx_location_road_{{ partition }}_geometry
ON location_road_{{ partition }}
USING GIST (geometry) {{db.tablespace.address_index}};
CREATE UNIQUE INDEX idx_location_road_{{ partition }}_place_id
ON location_road_{{ partition }}
USING BTREE (place_id) {{db.tablespace.address_index}};
CREATE INDEX idx_location_road_{{ partition }}_geometry ON location_road_{{ partition }} USING GIST (geometry) {{db.tablespace.address_index}};
CREATE INDEX idx_location_road_{{ partition }}_place_id ON location_road_{{ partition }} USING BTREE (place_id) {{db.tablespace.address_index}};
{% endfor %}

View File

@@ -2,7 +2,7 @@
--
-- This file is part of Nominatim. (https://nominatim.org)
--
-- Copyright (C) 2025 by the Nominatim developer community.
-- Copyright (C) 2022 by the Nominatim developer community.
-- For a full list of authors see the git log.
-- insert creates the location tables, creates location indexes if indexed == true
@@ -25,9 +25,5 @@ CREATE TRIGGER place_before_delete BEFORE DELETE ON place
CREATE TRIGGER place_before_insert BEFORE INSERT ON place
FOR EACH ROW EXECUTE PROCEDURE place_insert();
CREATE TRIGGER location_postcode_before_update BEFORE UPDATE ON location_postcodes
FOR EACH ROW EXECUTE PROCEDURE postcodes_update();
CREATE TRIGGER location_postcodes_before_delete BEFORE DELETE ON location_postcodes
FOR EACH ROW EXECUTE PROCEDURE postcodes_delete();
CREATE TRIGGER location_postcodes_before_insert BEFORE INSERT ON location_postcodes
FOR EACH ROW EXECUTE PROCEDURE postcodes_insert();
CREATE TRIGGER location_postcode_before_update BEFORE UPDATE ON location_postcode
FOR EACH ROW EXECUTE PROCEDURE postcode_update();

View File

@@ -2,7 +2,7 @@
--
-- This file is part of Nominatim. (https://nominatim.org)
--
-- Copyright (C) 2026 by the Nominatim developer community.
-- Copyright (C) 2022 by the Nominatim developer community.
-- For a full list of authors see the git log.
drop table if exists import_status;
@@ -11,6 +11,7 @@ CREATE TABLE import_status (
sequence_id integer,
indexed boolean
);
GRANT SELECT ON import_status TO "{{config.DATABASE_WEBUSER}}" ;
drop table if exists import_osmosis_log;
CREATE TABLE import_osmosis_log (
@@ -22,60 +23,82 @@ CREATE TABLE import_osmosis_log (
event text
);
CREATE TABLE new_query_log (
type text,
starttime timestamp,
ipaddress text,
useragent text,
language text,
query text,
searchterm text,
endtime timestamp,
results integer,
format text,
secret text
);
CREATE INDEX idx_new_query_log_starttime ON new_query_log USING BTREE (starttime);
GRANT INSERT ON new_query_log TO "{{config.DATABASE_WEBUSER}}" ;
GRANT UPDATE ON new_query_log TO "{{config.DATABASE_WEBUSER}}" ;
GRANT SELECT ON new_query_log TO "{{config.DATABASE_WEBUSER}}" ;
GRANT SELECT ON TABLE country_name TO "{{config.DATABASE_WEBUSER}}";
DROP TABLE IF EXISTS nominatim_properties;
CREATE TABLE nominatim_properties (
property TEXT NOT NULL,
value TEXT
);
GRANT SELECT ON TABLE nominatim_properties TO "{{config.DATABASE_WEBUSER}}";
drop table IF EXISTS location_area CASCADE;
CREATE TABLE location_area (
place_id BIGINT NOT NULL,
keywords INTEGER[] NOT NULL,
partition SMALLINT NOT NULL,
place_id BIGINT,
keywords INTEGER[],
partition SMALLINT,
rank_search SMALLINT NOT NULL,
rank_address SMALLINT NOT NULL,
country_code VARCHAR(2),
isguess BOOL NOT NULL,
isguess BOOL,
postcode TEXT,
centroid GEOMETRY(Point, 4326) NOT NULL,
geometry GEOMETRY(Geometry, 4326) NOT NULL
centroid GEOMETRY(Point, 4326),
geometry GEOMETRY(Geometry, 4326)
);
CREATE TABLE location_area_large () INHERITS (location_area);
DROP TABLE IF EXISTS location_area_country;
CREATE TABLE location_area_country (
place_id BIGINT NOT NULL,
country_code varchar(2) NOT NULL,
geometry GEOMETRY(Geometry, 4326) NOT NULL
place_id BIGINT,
country_code varchar(2),
geometry GEOMETRY(Geometry, 4326)
) {{db.tablespace.address_data}};
CREATE INDEX idx_location_area_country_geometry ON location_area_country USING GIST (geometry) {{db.tablespace.address_index}};
CREATE TABLE location_property_tiger (
place_id BIGINT NOT NULL,
place_id BIGINT,
parent_place_id BIGINT,
startnumber INTEGER NOT NULL,
endnumber INTEGER NOT NULL,
step SMALLINT NOT NULL,
partition SMALLINT NOT NULL,
linegeo GEOMETRY NOT NULL,
startnumber INTEGER,
endnumber INTEGER,
step SMALLINT,
partition SMALLINT,
linegeo GEOMETRY,
postcode TEXT);
GRANT SELECT ON location_property_tiger TO "{{config.DATABASE_WEBUSER}}";
drop table if exists location_property_osmline;
CREATE TABLE location_property_osmline (
place_id BIGINT NOT NULL,
osm_id BIGINT NOT NULL,
osm_id BIGINT,
parent_place_id BIGINT,
geometry_sector INTEGER NOT NULL,
geometry_sector INTEGER,
indexed_date TIMESTAMP,
startnumber INTEGER,
endnumber INTEGER,
step SMALLINT,
partition SMALLINT NOT NULL,
indexed_status SMALLINT NOT NULL,
linegeo GEOMETRY NOT NULL,
partition SMALLINT,
indexed_status SMALLINT,
linegeo GEOMETRY,
address HSTORE,
token_info JSONB, -- custom column for tokenizer use only
postcode TEXT,
@@ -85,31 +108,32 @@ CREATE UNIQUE INDEX idx_osmline_place_id ON location_property_osmline USING BTRE
CREATE INDEX idx_osmline_geometry_sector ON location_property_osmline USING BTREE (geometry_sector) {{db.tablespace.address_index}};
CREATE INDEX idx_osmline_linegeo ON location_property_osmline USING GIST (linegeo) {{db.tablespace.search_index}}
WHERE startnumber is not null;
GRANT SELECT ON location_property_osmline TO "{{config.DATABASE_WEBUSER}}";
drop table IF EXISTS search_name;
{% if not db.reverse_only %}
CREATE TABLE search_name (
place_id BIGINT NOT NULL,
importance FLOAT NOT NULL,
search_rank SMALLINT NOT NULL,
address_rank SMALLINT NOT NULL,
name_vector integer[] NOT NULL,
nameaddress_vector integer[] NOT NULL,
place_id BIGINT,
importance FLOAT,
search_rank SMALLINT,
address_rank SMALLINT,
name_vector integer[],
nameaddress_vector integer[],
country_code varchar(2),
centroid GEOMETRY(Geometry, 4326) NOT NULL
centroid GEOMETRY(Geometry, 4326)
) {{db.tablespace.search_data}};
CREATE UNIQUE INDEX idx_search_name_place_id
ON search_name USING BTREE (place_id) {{db.tablespace.search_index}};
CREATE INDEX idx_search_name_place_id ON search_name USING BTREE (place_id) {{db.tablespace.search_index}};
GRANT SELECT ON search_name to "{{config.DATABASE_WEBUSER}}" ;
{% endif %}
drop table IF EXISTS place_addressline;
CREATE TABLE place_addressline (
place_id BIGINT NOT NULL,
address_place_id BIGINT NOT NULL,
distance FLOAT NOT NULL,
cached_rank_address SMALLINT NOT NULL,
fromarea boolean NOT NULL,
isaddress boolean NOT NULL
place_id BIGINT,
address_place_id BIGINT,
distance FLOAT,
cached_rank_address SMALLINT,
fromarea boolean,
isaddress boolean
) {{db.tablespace.search_data}};
CREATE INDEX idx_place_addressline_place_id on place_addressline USING BTREE (place_id) {{db.tablespace.search_index}};
@@ -122,18 +146,18 @@ CREATE TABLE placex (
linked_place_id BIGINT,
importance FLOAT,
indexed_date TIMESTAMP,
geometry_sector INTEGER NOT NULL,
rank_address SMALLINT NOT NULL,
rank_search SMALLINT NOT NULL,
partition SMALLINT NOT NULL,
indexed_status SMALLINT NOT NULL,
geometry_sector INTEGER,
rank_address SMALLINT,
rank_search SMALLINT,
partition SMALLINT,
indexed_status SMALLINT,
LIKE place INCLUDING CONSTRAINTS,
wikipedia TEXT, -- calculated wikipedia article name (language:title)
token_info JSONB, -- custom column for tokenizer use only
country_code varchar(2),
housenumber TEXT,
postcode TEXT,
centroid GEOMETRY(Geometry, 4326) NOT NULL
centroid GEOMETRY(Geometry, 4326)
) {{db.tablespace.search_data}};
CREATE UNIQUE INDEX idx_place_id ON placex USING BTREE (place_id) {{db.tablespace.search_index}};
@@ -168,7 +192,8 @@ CREATE INDEX idx_placex_geometry_buildings ON placex
-- - linking of place nodes with same type to boundaries
CREATE INDEX idx_placex_geometry_placenode ON placex
USING SPGIST (geometry) {{db.tablespace.address_index}}
WHERE osm_type = 'N' and rank_search < 26 and class = 'place';
WHERE osm_type = 'N' and rank_search < 26
and class = 'place' and type != 'postcode';
-- Usage: - is node part of a way?
-- - find parent of interpolation spatially
@@ -196,30 +221,28 @@ CREATE INDEX idx_placex_rank_boundaries_sector ON placex
DROP SEQUENCE IF EXISTS seq_place;
CREATE SEQUENCE seq_place start 1;
GRANT SELECT on placex to "{{config.DATABASE_WEBUSER}}" ;
GRANT SELECT on place_addressline to "{{config.DATABASE_WEBUSER}}" ;
GRANT SELECT ON planet_osm_ways to "{{config.DATABASE_WEBUSER}}" ;
GRANT SELECT ON planet_osm_rels to "{{config.DATABASE_WEBUSER}}" ;
GRANT SELECT on location_area to "{{config.DATABASE_WEBUSER}}" ;
-- Table for synthetic postcodes.
DROP TABLE IF EXISTS location_postcodes;
CREATE TABLE location_postcodes (
place_id BIGINT NOT NULL,
DROP TABLE IF EXISTS location_postcode;
CREATE TABLE location_postcode (
place_id BIGINT,
parent_place_id BIGINT,
osm_id BIGINT,
rank_search SMALLINT NOT NULL,
indexed_status SMALLINT NOT NULL,
rank_search SMALLINT,
rank_address SMALLINT,
indexed_status SMALLINT,
indexed_date TIMESTAMP,
country_code varchar(2) NOT NULL,
postcode TEXT NOT NULL,
centroid GEOMETRY(Geometry, 4326) NOT NULL,
geometry GEOMETRY(Geometry, 4326) NOT NULL
country_code varchar(2),
postcode TEXT,
geometry GEOMETRY(Geometry, 4326)
);
CREATE UNIQUE INDEX idx_location_postcodes_id ON location_postcodes
USING BTREE (place_id) {{db.tablespace.search_index}};
CREATE INDEX idx_location_postcodes_geometry ON location_postcodes
USING GIST (geometry) {{db.tablespace.search_index}};
CREATE INDEX IF NOT EXISTS idx_location_postcodes_postcode
ON location_postcodes USING BTREE (postcode, country_code)
{{db.tablespace.search_index}};
CREATE INDEX IF NOT EXISTS idx_location_postcodes_osmid
ON location_postcodes USING BTREE (osm_id) {{db.tablespace.search_index}};
CREATE UNIQUE INDEX idx_postcode_id ON location_postcode USING BTREE (place_id) {{db.tablespace.search_index}};
CREATE INDEX idx_postcode_geometry ON location_postcode USING GIST (geometry) {{db.tablespace.address_index}};
GRANT SELECT ON location_postcode TO "{{config.DATABASE_WEBUSER}}" ;
-- Table to store location of entrance nodes
DROP TABLE IF EXISTS placex_entrance;
@@ -232,6 +255,7 @@ CREATE TABLE placex_entrance (
);
CREATE UNIQUE INDEX idx_placex_entrance_place_id_osm_id ON placex_entrance
USING BTREE (place_id, osm_id) {{db.tablespace.search_index}};
GRANT SELECT ON placex_entrance TO "{{config.DATABASE_WEBUSER}}" ;
-- Create an index on the place table for lookups to populate the entrance
-- table
@@ -253,6 +277,7 @@ CREATE TABLE import_polygon_error (
newgeometry GEOMETRY(Geometry, 4326)
);
CREATE INDEX idx_import_polygon_error_osmid ON import_polygon_error USING BTREE (osm_type, osm_id);
GRANT SELECT ON import_polygon_error TO "{{config.DATABASE_WEBUSER}}";
DROP TABLE IF EXISTS import_polygon_delete;
CREATE TABLE import_polygon_delete (
@@ -262,6 +287,7 @@ CREATE TABLE import_polygon_delete (
type TEXT NOT NULL
);
CREATE INDEX idx_import_polygon_delete_osmid ON import_polygon_delete USING BTREE (osm_type, osm_id);
GRANT SELECT ON import_polygon_delete TO "{{config.DATABASE_WEBUSER}}";
DROP SEQUENCE IF EXISTS file;
CREATE SEQUENCE file start 1;
@@ -292,3 +318,5 @@ CREATE INDEX planet_osm_rels_relation_members_idx ON planet_osm_rels USING gin(p
CREATE INDEX IF NOT EXISTS idx_place_interpolations
ON place USING gist(geometry) {{db.tablespace.address_index}}
WHERE osm_type = 'W' and address ? 'interpolation';
GRANT SELECT ON table country_osm_grid to "{{config.DATABASE_WEBUSER}}";

View File

@@ -15,99 +15,6 @@ CREATE TABLE location_property_tiger_import (
step SMALLINT,
postcode TEXT);
-- Lookup functions for tiger import when update
-- tables are dropped (see gh-issue #2463)
CREATE OR REPLACE FUNCTION getNearestNamedRoadPlaceIdSlow(in_centroid GEOMETRY,
in_token_info JSONB)
RETURNS BIGINT
AS $$
DECLARE
out_place_id BIGINT;
BEGIN
SELECT place_id INTO out_place_id
FROM search_name
WHERE
-- finds rows where name_vector shares elements with search tokens.
token_matches_street(in_token_info, name_vector)
-- limits search area
AND centroid && ST_Expand(in_centroid, 0.015)
AND address_rank BETWEEN 26 AND 27
ORDER BY ST_Distance(centroid, in_centroid) ASC
LIMIT 1;
RETURN out_place_id;
END
$$
LANGUAGE plpgsql;
CREATE OR REPLACE FUNCTION getNearestParallelRoadFeatureSlow(line GEOMETRY)
RETURNS BIGINT
AS $$
DECLARE
r RECORD;
search_diameter FLOAT;
p1 GEOMETRY;
p2 GEOMETRY;
p3 GEOMETRY;
BEGIN
IF ST_GeometryType(line) not in ('ST_LineString') THEN
RETURN NULL;
END IF;
p1 := ST_LineInterpolatePoint(line,0);
p2 := ST_LineInterpolatePoint(line,0.5);
p3 := ST_LineInterpolatePoint(line,1);
search_diameter := 0.0005;
WHILE search_diameter < 0.01 LOOP
FOR r IN
SELECT place_id FROM placex
WHERE ST_DWithin(line, geometry, search_diameter)
AND rank_address BETWEEN 26 AND 27
ORDER BY (ST_distance(geometry, p1)+
ST_distance(geometry, p2)+
ST_distance(geometry, p3)) ASC limit 1
LOOP
RETURN r.place_id;
END LOOP;
search_diameter := search_diameter * 2;
END LOOP;
RETURN NULL;
END
$$
LANGUAGE plpgsql;
CREATE OR REPLACE FUNCTION getNearestRoadPlaceIdSlow(point GEOMETRY)
RETURNS BIGINT
AS $$
DECLARE
r RECORD;
search_diameter FLOAT;
BEGIN
search_diameter := 0.00005;
WHILE search_diameter < 0.1 LOOP
FOR r IN
SELECT place_id FROM placex
WHERE ST_DWithin(geometry, point, search_diameter)
AND rank_address BETWEEN 26 AND 27
ORDER BY ST_Distance(geometry, point) ASC limit 1
LOOP
RETURN r.place_id;
END LOOP;
search_diameter := search_diameter * 2;
END LOOP;
RETURN NULL;
END
$$
LANGUAGE plpgsql;
-- Tiger import function
CREATE OR REPLACE FUNCTION tiger_line_import(linegeo GEOMETRY, in_startnumber INTEGER,
in_endnumber INTEGER, interpolationtype TEXT,
token_info JSONB, in_postcode TEXT) RETURNS INTEGER
@@ -164,51 +71,28 @@ BEGIN
place_centroid := ST_Centroid(linegeo);
out_partition := get_partition('us');
-- HYBRID LOOKUP LOGIC (see gh-issue #2463)
-- if partition tables exist, use them for fast spatial lookups
{% if 'location_road_0' in db.tables %}
out_parent_place_id := getNearestNamedRoadPlaceId(out_partition, place_centroid,
out_parent_place_id := getNearestNamedRoadPlaceId(out_partition, place_centroid,
token_info);
IF out_parent_place_id IS NULL THEN
SELECT getNearestParallelRoadFeature(out_partition, linegeo)
INTO out_parent_place_id;
END IF;
IF out_parent_place_id IS NULL THEN
SELECT getNearestRoadPlaceId(out_partition, place_centroid)
INTO out_parent_place_id;
END IF;
-- When updatable information has been dropped:
-- Partition tables no longer exist, but search_name still persists.
{% elif 'search_name' in db.tables %}
-- Fallback: Look up in 'search_name' table
-- though spatial lookups here can be slower.
out_parent_place_id := getNearestNamedRoadPlaceIdSlow(place_centroid, token_info);
IF out_parent_place_id IS NULL THEN
out_parent_place_id := getNearestParallelRoadFeatureSlow(linegeo);
END IF;
IF out_parent_place_id IS NULL THEN
out_parent_place_id := getNearestRoadPlaceIdSlow(place_centroid);
END IF;
{% endif %}
-- If parent was found, insert street(line) into import table
IF out_parent_place_id IS NOT NULL THEN
INSERT INTO location_property_tiger_import (linegeo, place_id, partition,
parent_place_id, startnumber, endnumber,
step, postcode)
VALUES (linegeo, nextval('seq_place'), out_partition,
out_parent_place_id, startnumber, endnumber,
stepsize, in_postcode);
RETURN 1;
IF out_parent_place_id IS NULL THEN
SELECT getNearestParallelRoadFeature(out_partition, linegeo)
INTO out_parent_place_id;
END IF;
RETURN 0;
IF out_parent_place_id IS NULL THEN
SELECT getNearestRoadPlaceId(out_partition, place_centroid)
INTO out_parent_place_id;
END IF;
--insert street(line) into import table
insert into location_property_tiger_import (linegeo, place_id, partition,
parent_place_id, startnumber, endnumber,
step, postcode)
values (linegeo, nextval('seq_place'), out_partition,
out_parent_place_id, startnumber, endnumber,
stepsize, in_postcode);
RETURN 1;
END;
$$
LANGUAGE plpgsql;

View File

@@ -1,4 +1,4 @@
site_name: Nominatim Manual
site_name: Nominatim 5.2.0 Manual
theme:
font: false
name: material

View File

@@ -23,7 +23,7 @@ an ASGI-capable server like uvicorn. To install them from pypi run:
You need to have a Nominatim database imported with the 'nominatim-db'
package. Go to the project directory, then run uvicorn as:
uvicorn --factory nominatim_api.server.falcon.server:run_wsgi
uvicorn --factory nominatim.server.falcon.server:run_wsgi
## Documentation

View File

@@ -15,13 +15,12 @@ classifiers = [
"Operating System :: OS Independent",
]
dependencies = [
"psycopg != 3.3.0",
"psycopg",
"python-dotenv",
"jinja2",
"pyYAML>=5.1",
"psutil",
"PyICU",
"mwparserfromhell"
"PyICU"
]
dynamic = ["version"]

View File

@@ -1,4 +1,3 @@
name:
default: الأراضي الفلسطينية
en: Palestinian Territories
name:
default: Palestinian Territory
"no": Det palestinske området

View File

@@ -305,7 +305,6 @@ ch:
names: !include country-names/ch.yaml
postcode:
pattern: "dddd"
extent: 3000
# Côte d'Ivoire (Côte dIvoire)
@@ -473,7 +472,6 @@ ee:
names: !include country-names/ee.yaml
postcode:
pattern: "ddddd"
extent: 3000
# Egypt (مصر)
@@ -587,7 +585,6 @@ gb:
postcode:
pattern: "(l?ld[A-Z0-9]?) ?(dll)"
output: \1 \2
extent: 700
# Grenada (Grenada)
@@ -615,7 +612,6 @@ gg:
postcode:
pattern: "(GYdd?) ?(dll)"
output: \1 \2
extent: 1000
# Ghana (Ghana)
@@ -770,7 +766,6 @@ ie:
postcode:
pattern: "(ldd) ?([0123456789ACDEFHKNPRTVWXY]{4})"
output: \1 \2
extent: 50
# Israel (ישראל)
@@ -790,7 +785,6 @@ im:
postcode:
pattern: "(IMdd?) ?(dll)"
output: \1 \2
extent: 700
# India (India)
@@ -885,7 +879,6 @@ jp:
postcode:
pattern: "(ddd)-?(dddd)"
output: \1-\2
extent: 3000
# Kenya (Kenya)
@@ -1020,7 +1013,6 @@ li:
names: !include country-names/li.yaml
postcode:
pattern: "dddd"
extent: 4000
# Sri Lanka (ශ්‍රී ලංකාව இலங்கை)
@@ -1066,7 +1058,6 @@ lu:
names: !include country-names/lu.yaml
postcode:
pattern: "dddd"
extent: 1000
# Latvia (Latvija)
@@ -1299,7 +1290,6 @@ nl:
postcode:
pattern: "(dddd) ?(ll)"
output: \1 \2
extent: 800
# Norway (Norge)
@@ -1435,7 +1425,6 @@ pt:
names: !include country-names/pt.yaml
postcode:
pattern: "dddd(?:-ddd)?"
extent: 1000
# Palau (Belau)
@@ -1471,7 +1460,6 @@ ro:
names: !include country-names/ro.yaml
postcode:
pattern: "dddddd"
extent: 2500
# Serbia (Србија)

View File

@@ -8,6 +8,7 @@
Helper functions for localizing names of results.
"""
from typing import Mapping, List, Optional
from .config import Configuration
from .results import AddressLines, BaseResultT
import re
@@ -17,15 +18,15 @@ class Locales:
""" Helper class for localization of names.
It takes a list of language prefixes in their order of preferred
usage and comma separated name keys (Configuration.OUTPUT_NAMES).
usage.
"""
def __init__(self, langs: Optional[List[str]] = None,
names: str = 'name:XX,name') -> None:
def __init__(self, langs: Optional[List[str]] = None):
self.config = Configuration(None)
self.languages = langs or []
self.name_tags: List[str] = []
parts = names.split(',') if names else []
parts = self.config.OUTPUT_NAMES.split(',')
for part in parts:
part = part.strip()
@@ -67,7 +68,7 @@ class Locales:
return next(iter(names.values()))
@staticmethod
def from_accept_languages(langstr: str, names: str = 'name:XX,name') -> 'Locales':
def from_accept_languages(langstr: str) -> 'Locales':
""" Create a localization object from a language list in the
format of HTTP accept-languages header.
@@ -95,7 +96,7 @@ class Locales:
if len(parts) > 1 and all(c[0] != parts[0] for c in candidates):
languages.append(parts[0])
return Locales(languages, names)
return Locales(languages)
def localize(self, lines: AddressLines) -> None:
""" Sets the local name of address parts according to the chosen

View File

@@ -288,7 +288,7 @@ class TextLogger(BaseLogger):
self._write(f"rank={res.rank_address}, ")
self._write(f"osm={''.join(map(str, res.osm_object or []))}, ")
self._write(f'cc={res.country_code}, ')
self._write(f'importance={res.importance or float("NaN"):.5f})\n')
self._write(f'importance={res.importance or -1:.5f})\n')
total += 1
self._write(f'TOTAL: {total}\n\n')

View File

@@ -2,7 +2,7 @@
#
# This file is part of Nominatim. (https://nominatim.org)
#
# Copyright (C) 2025 by the Nominatim developer community.
# Copyright (C) 2024 by the Nominatim developer community.
# For a full list of authors see the git log.
"""
Implementation of place lookup by ID (doing many places at once).
@@ -291,30 +291,12 @@ async def find_in_postcode(conn: SearchConnection, collector: Collector) -> bool
.table_valued(sa.column('value', type_=sa.JSON))
t = conn.t.postcode
sql = sa.select(pid_tab.c.value['i'].as_integer().label('_idx'),
t.c.osm_id, t.c.place_id, t.c.parent_place_id,
t.c.rank_search,
t.c.place_id, t.c.parent_place_id,
t.c.rank_search, t.c.rank_address,
t.c.indexed_date, t.c.postcode, t.c.country_code,
t.c.centroid)\
t.c.geometry.label('centroid'))\
.where(t.c.place_id == pid_tab.c.value['id'].as_string().cast(sa.BigInteger))
if await collector.add_rows_from_sql(conn, sql, t.c.geometry,
nres.create_from_postcode_row):
return True
osm_ids = [{'i': i, 'oi': p.osm_id}
for i, p in collector.enumerate_free_osm_ids() if p.osm_type == 'R']
if osm_ids:
pid_tab = sa.func.JsonArrayEach(sa.type_coerce(osm_ids, sa.JSON))\
.table_valued(sa.column('value', type_=sa.JSON))
t = conn.t.postcode
sql = sa.select(pid_tab.c.value['i'].as_integer().label('_idx'),
t.c.osm_id, t.c.place_id, t.c.parent_place_id,
t.c.rank_search,
t.c.indexed_date, t.c.postcode, t.c.country_code,
t.c.centroid)\
.where(t.c.osm_id == pid_tab.c.value['oi'].as_string().cast(sa.BigInteger))
return await collector.add_rows_from_sql(conn, sql, t.c.geometry,
nres.create_from_postcode_row)

View File

@@ -10,7 +10,7 @@ Helper classes and functions for formatting results into API responses.
from typing import Type, TypeVar, Dict, List, Callable, Any, Mapping, Optional, cast
from collections import defaultdict
from pathlib import Path
import importlib.util
import importlib
from .server.content_types import CONTENT_JSON
@@ -43,7 +43,7 @@ class FormatDispatcher:
return decorator
def error_format_func(self, func: ErrorFormatFunc) -> ErrorFormatFunc:
""" Decorator for a function that formats error messages.
""" Decorator for a function that formats error messges.
There is only one error formatter per dispatcher. Using
the decorator repeatedly will overwrite previous functions.
"""
@@ -79,7 +79,7 @@ class FormatDispatcher:
def set_content_type(self, fmt: str, content_type: str) -> None:
""" Set the content type for the given format. This is the string
that will be returned in the Content-Type header of the HTML
response, when the given format is chosen.
response, when the given format is choosen.
"""
self.content_types[fmt] = content_type

View File

@@ -2,7 +2,7 @@
#
# This file is part of Nominatim. (https://nominatim.org)
#
# Copyright (C) 2025 by the Nominatim developer community.
# Copyright (C) 2024 by the Nominatim developer community.
# For a full list of authors see the git log.
"""
Dataclasses for search results and helper functions to fill them.
@@ -407,13 +407,11 @@ def create_from_postcode_row(row: SaRow, class_type: Type[BaseResultT]) -> BaseR
"""
return class_type(source_table=SourceTable.POSTCODE,
place_id=row.place_id,
osm_object=None if row.osm_id is None else ('R', row.osm_id),
parent_place_id=row.parent_place_id,
category=(('place', 'postcode') if row.osm_id is None
else ('boundary', 'postal_code')),
category=('place', 'postcode'),
names={'ref': row.postcode},
rank_search=row.rank_search,
rank_address=5,
rank_address=row.rank_address,
country_code=row.country_code,
centroid=Point.from_wkb(row.centroid),
geometry=_filter_geometries(row))
@@ -496,15 +494,17 @@ def _get_address_lookup_id(result: BaseResultT) -> int:
async def _finalize_entry(conn: SearchConnection, result: BaseResultT) -> None:
assert result.address_rows is not None
postcode = result.postcode or (result.address and result.address.get('postcode'))
if postcode and ',' not in postcode and ';' not in postcode:
result.address_rows.append(AddressLine(
category=('place', 'postcode'),
names={'ref': postcode},
fromarea=False, isaddress=True, rank_address=5,
distance=0.0))
if result.category[0] not in ('boundary', 'place')\
or result.category[1] not in ('postal_code', 'postcode'):
postcode = result.postcode
if not postcode and result.address:
postcode = result.address.get('postcode')
if postcode and ',' not in postcode and ';' not in postcode:
result.address_rows.append(AddressLine(
category=('place', 'postcode'),
names={'ref': postcode},
fromarea=False, isaddress=True, rank_address=5,
distance=0.0))
if result.country_code:
async def _get_country_names() -> Optional[Dict[str, str]]:
t = conn.t.country_name
@@ -627,6 +627,13 @@ async def complete_address_details(conn: SearchConnection, results: List[BaseRes
if current_result.country_code is None and row.country_code:
current_result.country_code = row.country_code
if row.type in ('postcode', 'postal_code') and location_isaddress:
if not row.fromarea or \
(current_result.address and 'postcode' in current_result.address):
location_isaddress = False
else:
current_result.postcode = None
assert current_result.address_rows is not None
current_result.address_rows.append(_result_row_to_address_row(row, location_isaddress))
current_rank_address = row.rank_address

View File

@@ -157,19 +157,16 @@ class ReverseGeocoder:
include.extend(('natural', 'water', 'waterway'))
return table.c.class_.in_(tuple(include))
async def _find_closest_street_or_pois(self, distance: float,
fuzziness: float) -> list[SaRow]:
""" Look up the closest rank 26+ place in the database.
The function finds the object that is closest to the reverse
search point as well as all objects within 'fuzziness' distance
to that best result.
async def _find_closest_street_or_poi(self, distance: float) -> Optional[SaRow]:
""" Look up the closest rank 26+ place in the database, which
is closer than the given distance.
"""
t = self.conn.t.placex
# PostgreSQL must not get the distance as a parameter because
# there is a danger it won't be able to properly estimate index use
# when used with prepared statements
diststr = sa.text(f"{distance + fuzziness}")
diststr = sa.text(f"{distance}")
sql: SaLambdaSelect = sa.lambda_stmt(
lambda: _select_from_placex(t)
@@ -177,7 +174,9 @@ class ReverseGeocoder:
.where(t.c.indexed_status == 0)
.where(t.c.linked_place_id == None)
.where(sa.or_(sa.not_(t.c.geometry.is_area()),
t.c.centroid.ST_Distance(WKT_PARAM) < diststr)))
t.c.centroid.ST_Distance(WKT_PARAM) < diststr))
.order_by('distance')
.limit(2))
if self.has_geometries():
sql = self._add_geometry_columns(sql, t.c.geometry)
@@ -199,44 +198,24 @@ class ReverseGeocoder:
self._filter_by_layer(t)))
if not restrict:
return []
return None
inner = sql.where(sa.or_(*restrict)) \
.add_columns(t.c.geometry.label('_geometry')) \
.subquery()
sql = sql.where(sa.or_(*restrict))
# Use a window function to get the closest results to the best result.
windowed = sa.select(inner,
sa.func.first_value(inner.c.distance)
.over(order_by=inner.c.distance)
.label('_min_distance'),
sa.func.first_value(
sa.case((inner.c.rank_search <= 27,
inner.c._geometry.ST_ClosestPoint(WKT_PARAM)),
else_=None))
.over(order_by=inner.c.distance)
.label('_closest_point'),
sa.func.first_value(sa.case((sa.or_(inner.c.rank_search <= 27,
inner.c.osm_type == 'N'), None),
else_=inner.c._geometry))
.over(order_by=inner.c.distance)
.label('_best_geometry')) \
.subquery()
# If the closest object is inside an area, then check if there is a
# POI node nearby and return that.
prev_row = None
for row in await self.conn.execute(sql, self.bind_params):
if prev_row is None:
if row.rank_search <= 27 or row.osm_type == 'N' or row.distance > 0:
return row
prev_row = row
else:
if row.rank_search > 27 and row.osm_type == 'N'\
and row.distance < 0.0001:
return row
outer = sa.select(*(c for c in windowed.c if not c.key.startswith('_')),
sa.case((sa.or_(windowed.c._closest_point == None,
windowed.c.housenumber == None), None),
else_=windowed.c.centroid.ST_Distance(windowed.c._closest_point))
.label('distance_from_best'),
sa.case((sa.or_(windowed.c._best_geometry == None,
windowed.c.rank_search <= 27,
windowed.c.osm_type != 'N'), False),
else_=windowed.c.centroid.ST_CoveredBy(windowed.c._best_geometry))
.label('best_inside')) \
.where(windowed.c.distance < windowed.c._min_distance + fuzziness) \
.order_by(windowed.c.distance)
return list(await self.conn.execute(outer, self.bind_params))
return prev_row
async def _find_housenumber_for_street(self, parent_place_id: int) -> Optional[SaRow]:
t = self.conn.t.placex
@@ -322,69 +301,55 @@ class ReverseGeocoder:
""" Find a street or POI/address for the given WKT point.
"""
log().section('Reverse lookup on street/address level')
row_func: RowFunc = nres.create_from_placex_row
distance = 0.006
parent_place_id = None
result = None
hnr_distance = None
parent_street = None
for row in await self._find_closest_street_or_pois(distance, 0.001):
if result is None:
log().var_dump('Closest result', row)
result = row
if self.max_rank > 27 \
and self.layer_enabled(DataLayer.ADDRESS) \
and result.rank_address <= 27:
parent_street = result.place_id
distance = 0.001
else:
distance = row.distance
# If the closest result was a street but an address was requested,
# see if we can refine the result with a housenumber closeby.
elif parent_street is not None \
and row.distance_from_best is not None \
and row.distance_from_best < 0.001 \
and (hnr_distance is None or hnr_distance > row.distance_from_best) \
and row.parent_place_id == parent_street:
log().var_dump('Housenumber to closest result', row)
result = row
hnr_distance = row.distance_from_best
row = await self._find_closest_street_or_poi(distance)
row_func: RowFunc = nres.create_from_placex_row
log().var_dump('Result (street/building)', row)
# If the closest result was a street, but an address was requested,
# check for a housenumber nearby which is part of the street.
if row is not None:
if self.max_rank > 27 \
and self.layer_enabled(DataLayer.ADDRESS) \
and row.rank_address <= 27:
distance = 0.001
parent_place_id = row.place_id
log().comment('Find housenumber for street')
addr_row = await self._find_housenumber_for_street(parent_place_id)
log().var_dump('Result (street housenumber)', addr_row)
if addr_row is not None:
row = addr_row
row_func = nres.create_from_placex_row
distance = addr_row.distance
elif row.country_code == 'us' and parent_place_id is not None:
log().comment('Find TIGER housenumber for street')
addr_row = await self._find_tiger_number_for_street(parent_place_id)
log().var_dump('Result (street Tiger housenumber)', addr_row)
if addr_row is not None:
row_func = cast(RowFunc,
functools.partial(nres.create_from_tiger_row,
osm_type=row.osm_type,
osm_id=row.osm_id))
row = addr_row
else:
distance = row.distance
# If the closest object is inside an area, then check if there is
# a POI nearby and return that with preference.
elif result.osm_type != 'N' and result.rank_search > 27 \
and result.distance == 0 \
and row.best_inside:
log().var_dump('POI near closest result area', row)
result = row
break # it can't get better than that, everything else is farther away
# For the US also check the TIGER data, when no housenumber/POI was found.
if result is not None and parent_street is not None and hnr_distance is None \
and result.country_code == 'us':
log().comment('Find TIGER housenumber for street')
addr_row = await self._find_tiger_number_for_street(parent_street)
log().var_dump('Result (street Tiger housenumber)', addr_row)
if addr_row is not None:
row_func = cast(RowFunc,
functools.partial(nres.create_from_tiger_row,
osm_type=row.osm_type,
osm_id=row.osm_id))
result = addr_row
# Check for an interpolation that is either closer than our result
# or belongs to a close street found.
# No point in doing this when the result is already inside a building,
# i.e. when the distance is already 0.
if self.max_rank > 27 and self.layer_enabled(DataLayer.ADDRESS) and distance > 0:
if self.max_rank > 27 and self.layer_enabled(DataLayer.ADDRESS):
log().comment('Find interpolation for street')
addr_row = await self._find_interpolation_for_street(parent_street, distance)
addr_row = await self._find_interpolation_for_street(parent_place_id,
distance)
log().var_dump('Result (street interpolation)', addr_row)
if addr_row is not None:
return addr_row, nres.create_from_osmline_row
row = addr_row
row_func = nres.create_from_osmline_row
return result, row_func
return row, row_func
async def _lookup_area_address(self) -> Optional[SaRow]:
""" Lookup large addressable areas for the given WKT point.

View File

@@ -374,7 +374,7 @@ class SearchBuilder:
tokens = self.get_country_tokens(assignment.country)
if not tokens:
return None
sdata.set_countries(tokens)
sdata.set_strings('countries', tokens)
sdata.penalty += self.query.get_in_word_penalty(assignment.country)
elif self.details.countries:
sdata.countries = dbf.WeightedStrings(self.details.countries,
@@ -413,7 +413,7 @@ class SearchBuilder:
"""
tokens = self.query.get_tokens(trange, qmod.TOKEN_COUNTRY)
if self.details.countries:
tokens = [t for t in tokens if t.get_country() in self.details.countries]
tokens = [t for t in tokens if t.lookup_word in self.details.countries]
return tokens

View File

@@ -22,7 +22,7 @@ class CountedTokenIDs:
""" A list of token IDs with their respective counts, sorted
from least frequent to most frequent.
If a token count is one, then statistics are likely to be unavailable
If a token count is one, then statistics are likely to be unavaible
and a relatively high count is assumed instead.
"""
@@ -244,21 +244,6 @@ class SearchData:
setattr(self, field, wstrs)
def set_countries(self, tokens: List[Token]) -> None:
""" Set the WeightedStrings properties for countries. Multiple
entries for the same country are deduplicated and the minimum
penalty is used. Adapts the global penalty, so that the
minimum penalty is 0.
"""
if tokens:
min_penalty = min(t.penalty for t in tokens)
self.penalty += min_penalty
countries: dict[str, float] = {}
for t in tokens:
cc = t.get_country()
countries[cc] = min(t.penalty - min_penalty, countries.get(cc, 10000))
self.countries = WeightedStrings(list(countries.keys()), list(countries.values()))
def set_qualifiers(self, tokens: List[Token]) -> None:
""" Set the qulaifier field from the given tokens.
"""

View File

@@ -175,8 +175,7 @@ class AddressSearch(base.AbstractSearch):
sql = sql.where(sa.select(tpc.c.postcode)
.where(tpc.c.postcode.in_(self.postcodes.values))
.where(tpc.c.country_code == t.c.country_code)
.where(t.c.centroid.intersects(tpc.c.geometry,
use_index=False))
.where(t.c.centroid.within_distance(tpc.c.geometry, 0.4))
.exists())
if details.viewbox is not None:
@@ -226,7 +225,7 @@ class AddressSearch(base.AbstractSearch):
tpc = conn.t.postcode
pcs = self.postcodes.values
pc_near = sa.select(sa.func.min(tpc.c.centroid.ST_Distance(t.c.centroid)
pc_near = sa.select(sa.func.min(tpc.c.geometry.ST_Distance(t.c.centroid)
* (tpc.c.rank_search - 19)))\
.where(tpc.c.postcode.in_(pcs))\
.where(tpc.c.country_code == t.c.country_code)\

View File

@@ -79,8 +79,7 @@ class PlaceSearch(base.AbstractSearch):
tpc = conn.t.postcode
sql = sql.where(sa.select(tpc.c.postcode)
.where(tpc.c.postcode.in_(self.postcodes.values))
.where(t.c.centroid.intersects(tpc.c.geometry,
use_index=False))
.where(t.c.centroid.within_distance(tpc.c.geometry, 0.4))
.exists())
if details.viewbox is not None:
@@ -158,7 +157,7 @@ class PlaceSearch(base.AbstractSearch):
tpc = conn.t.postcode
pcs = self.postcodes.values
pc_near = sa.select(sa.func.min(tpc.c.centroid.ST_Distance(t.c.centroid)))\
pc_near = sa.select(sa.func.min(tpc.c.geometry.ST_Distance(t.c.centroid)))\
.where(tpc.c.postcode.in_(pcs))\
.scalar_subquery()
penalty += sa.case((t.c.postcode.in_(pcs), 0.0),

View File

@@ -14,7 +14,7 @@ from . import base
from ...typing import SaBind, SaExpression
from ...sql.sqlalchemy_types import Geometry, IntArray
from ...connection import SearchConnection
from ...types import SearchDetails
from ...types import SearchDetails, Bbox
from ... import results as nres
from ..db_search_fields import SearchData
@@ -42,9 +42,10 @@ class PostcodeSearch(base.AbstractSearch):
t = conn.t.postcode
pcs = self.postcodes.values
sql = sa.select(t.c.place_id, t.c.parent_place_id, t.c.osm_id,
t.c.rank_search, t.c.postcode, t.c.country_code,
t.c.centroid)\
sql = sa.select(t.c.place_id, t.c.parent_place_id,
t.c.rank_search, t.c.rank_address,
t.c.postcode, t.c.country_code,
t.c.geometry.label('centroid'))\
.where(t.c.postcode.in_(pcs))
if details.geometry_output:
@@ -58,7 +59,7 @@ class PostcodeSearch(base.AbstractSearch):
else_=1.0)
if details.near is not None:
sql = sql.order_by(t.c.centroid.ST_Distance(NEAR_PARAM))
sql = sql.order_by(t.c.geometry.ST_Distance(NEAR_PARAM))
sql = base.filter_by_area(sql, t, details)
@@ -99,9 +100,29 @@ class PostcodeSearch(base.AbstractSearch):
results = nres.SearchResults()
for row in await conn.execute(sql, bind_params):
result = nres.create_from_postcode_row(row, nres.SearchResult)
p = conn.t.placex
placex_sql = base.select_placex(p)\
.add_columns(p.c.importance)\
.where(sa.text("""class = 'boundary'
AND type = 'postal_code'
AND osm_type = 'R'"""))\
.where(p.c.country_code == row.country_code)\
.where(p.c.postcode == row.postcode)\
.limit(1)
result.accuracy = row.accuracy
results.append(result)
if details.geometry_output:
placex_sql = base.add_geometry_columns(placex_sql, p.c.geometry, details)
for prow in await conn.execute(placex_sql, bind_params):
result = nres.create_from_placex_row(prow, nres.SearchResult)
if result is not None:
result.bbox = Bbox.from_wkb(prow.bbox)
break
else:
result = nres.create_from_postcode_row(row, nres.SearchResult)
if result.place_id not in details.excluded:
result.accuracy = row.accuracy
results.append(result)
return results

View File

@@ -59,16 +59,12 @@ class ICUToken(qmod.Token):
assert self.info
return self.info.get('class', ''), self.info.get('type', '')
def get_country(self) -> str:
assert self.info
return cast(str, self.info.get('cc', ''))
def match_penalty(self, norm: str) -> float:
def rematch(self, norm: str) -> None:
""" Check how well the token matches the given normalized string
and add a penalty, if necessary.
"""
if not self.lookup_word:
return 0.0
return
seq = difflib.SequenceMatcher(a=self.lookup_word, b=norm)
distance = 0
@@ -79,7 +75,7 @@ class ICUToken(qmod.Token):
distance += max((ato-afrom), (bto-bfrom))
elif tag != 'equal':
distance += abs((ato-afrom) - (bto-bfrom))
return (distance/len(self.lookup_word))
self.penalty += (distance/len(self.lookup_word))
@staticmethod
def from_db_row(row: SaRow) -> 'ICUToken':
@@ -334,10 +330,9 @@ class ICUQueryAnalyzer(AbstractQueryAnalyzer):
norm = ''.join(f"{n.term_normalized}{'' if n.btype == qmod.BREAK_TOKEN else ' '}"
for n in query.nodes[start + 1:end + 1]).strip()
for ttype, tokens in tlist.items():
for token in tokens:
itok = cast(ICUToken, token)
itok.penalty += itok.match_penalty(norm) * \
(1 if ttype in (qmod.TOKEN_WORD, qmod.TOKEN_PARTIAL) else 2)
if ttype != qmod.TOKEN_COUNTRY:
for token in tokens:
cast(ICUToken, token).rematch(norm)
def compute_break_penalties(self, query: qmod.QueryStruct) -> None:
""" Set the break penalties for the nodes in the query.

View File

@@ -17,7 +17,7 @@ import dataclasses
# The x value for the regression computation will be the position of the
# token in the query. Thus we know the x values will be [0, query length).
# As the denominator only depends on the x values, we can pre-compute here
# the denominator to use for a given query length.
# the denominatior to use for a given query length.
# Note that query length of two or less is special cased and will not use
# the values from this array. Thus it is not a problem that they are 0.
LINFAC = [i * (sum(si * si for si in range(i)) - (i - 1) * i * (i - 1) / 4)
@@ -127,12 +127,6 @@ class Token(ABC):
category objects.
"""
@abstractmethod
def get_country(self) -> str:
""" Return the country code this token is associated with
(currently for country tokens only).
"""
@dataclasses.dataclass
class TokenRange:
@@ -231,7 +225,7 @@ class QueryNode:
return max(0, -self.penalty)
def name_address_ratio(self) -> float:
""" Return the probability that the partial token belonging to
""" Return the propability that the partial token belonging to
this node forms part of a name (as opposed of part of the address).
"""
if self.partial is None:
@@ -275,7 +269,7 @@ class QueryStruct:
directed acyclic graph.
A query also has a direction penalty 'dir_penalty'. This describes
the likelihood if the query should be read from left-to-right or
the likelyhood if the query should be read from left-to-right or
vice versa. A negative 'dir_penalty' should be read as a penalty on
right-to-left reading, while a positive value represents a penalty
for left-to-right reading. The default value is 0, which is equivalent

View File

@@ -7,8 +7,6 @@
"""
Server implementation using the falcon webserver framework.
"""
from __future__ import annotations
from typing import Optional, Mapping, Any, List, cast
from pathlib import Path
import asyncio
@@ -163,7 +161,7 @@ class APIMiddleware:
def __init__(self, project_dir: Path, environ: Optional[Mapping[str, str]]) -> None:
self.api = NominatimAPIAsync(project_dir, environ)
self.app: Optional[App[Request, Response]] = None
self.app: Optional[App] = None
@property
def config(self) -> Configuration:
@@ -171,7 +169,7 @@ class APIMiddleware:
"""
return self.api.config
def set_app(self, app: App[Request, Response]) -> None:
def set_app(self, app: App) -> None:
""" Set the Falcon application this middleware is connected to.
"""
self.app = app
@@ -195,7 +193,7 @@ class APIMiddleware:
def get_application(project_dir: Path,
environ: Optional[Mapping[str, str]] = None) -> App[Request, Response]:
environ: Optional[Mapping[str, str]] = None) -> App:
""" Create a Nominatim Falcon ASGI application.
"""
apimw = APIMiddleware(project_dir, environ)
@@ -217,7 +215,7 @@ def get_application(project_dir: Path,
return app
def run_wsgi() -> App[Request, Response]:
def run_wsgi() -> App:
""" Entry point for uvicorn.
Make sure uvicorn is run from the project directory.

View File

@@ -50,7 +50,7 @@ class ParamWrapper(ASGIAdaptor):
headers={'content-type': self.content_type})
def create_response(self, status: int, output: str, num_results: int) -> Response:
setattr(self.request.state, 'num_results', num_results)
self.request.state.num_results = num_results
return Response(output, status_code=status, media_type=self.content_type)
def base_uri(self) -> str:
@@ -95,7 +95,7 @@ class FileLoggingMiddleware(BaseHTTPMiddleware):
async def dispatch(self, request: Request,
call_next: RequestResponseEndpoint) -> Response:
qs = QueryStatistics()
setattr(request.state, 'query_stats', qs)
request.state.query_stats = qs
response = await call_next(request)
if response.status_code != 200 or 'start' not in qs:

View File

@@ -2,7 +2,7 @@
#
# This file is part of Nominatim. (https://nominatim.org)
#
# Copyright (C) 2025 by the Nominatim developer community.
# Copyright (C) 2024 by the Nominatim developer community.
# For a full list of authors see the git log.
"""
Custom functions and expressions for SQLAlchemy.
@@ -32,6 +32,7 @@ def _default_intersects(element: PlacexGeometryReverseLookuppolygon,
compiler: 'sa.Compiled', **kw: Any) -> str:
return ("(ST_GeometryType(placex.geometry) in ('ST_Polygon', 'ST_MultiPolygon')"
" AND placex.rank_address between 4 and 25"
" AND placex.type != 'postcode'"
" AND placex.name is not null"
" AND placex.indexed_status = 0"
" AND placex.linked_place_id is null)")
@@ -42,6 +43,7 @@ def _sqlite_intersects(element: PlacexGeometryReverseLookuppolygon,
compiler: 'sa.Compiled', **kw: Any) -> str:
return ("(ST_GeometryType(placex.geometry) in ('POLYGON', 'MULTIPOLYGON')"
" AND placex.rank_address between 4 and 25"
" AND placex.type != 'postcode'"
" AND placex.name is not null"
" AND placex.indexed_status = 0"
" AND placex.linked_place_id is null)")
@@ -62,6 +64,7 @@ def default_reverse_place_diameter(element: IntersectsReverseDistance,
compiler: 'sa.Compiled', **kw: Any) -> str:
table = element.tablename
return f"({table}.rank_address between 4 and 25"\
f" AND {table}.type != 'postcode'"\
f" AND {table}.name is not null"\
f" AND {table}.linked_place_id is null"\
f" AND {table}.osm_type = 'N'" + \
@@ -76,6 +79,7 @@ def sqlite_reverse_place_diameter(element: IntersectsReverseDistance,
table = element.tablename
return (f"({table}.rank_address between 4 and 25"
f" AND {table}.type != 'postcode'"
f" AND {table}.name is not null"
f" AND {table}.linked_place_id is null"
f" AND {table}.osm_type = 'N'"

View File

@@ -2,7 +2,7 @@
#
# This file is part of Nominatim. (https://nominatim.org)
#
# Copyright (C) 2025 by the Nominatim developer community.
# Copyright (C) 2024 by the Nominatim developer community.
# For a full list of authors see the git log.
"""
SQLAlchemy definitions for all tables used by the frontend.
@@ -67,16 +67,15 @@ class SearchTables:
sa.Column('isaddress', sa.Boolean))
self.postcode = sa.Table(
'location_postcodes', meta,
'location_postcode', meta,
sa.Column('place_id', sa.BigInteger),
sa.Column('parent_place_id', sa.BigInteger),
sa.Column('osm_id', sa.BigInteger),
sa.Column('rank_search', sa.SmallInteger),
sa.Column('rank_address', sa.SmallInteger),
sa.Column('indexed_status', sa.SmallInteger),
sa.Column('indexed_date', sa.DateTime),
sa.Column('country_code', sa.String(2)),
sa.Column('postcode', sa.Text),
sa.Column('centroid', Geometry),
sa.Column('geometry', Geometry))
self.osmline = sa.Table(

View File

@@ -2,7 +2,7 @@
#
# This file is part of Nominatim. (https://nominatim.org)
#
# Copyright (C) 2025 by the Nominatim developer community.
# Copyright (C) 2024 by the Nominatim developer community.
# For a full list of authors see the git log.
"""
Custom types for SQLAlchemy.
@@ -178,8 +178,6 @@ class Geometry(types.UserDefinedType): # type: ignore[type-arg]
def bind_processor(self, dialect: 'sa.Dialect') -> Callable[[Any], str]:
def process(value: Any) -> str:
if value is None:
return 'null'
if isinstance(value, str):
return value

View File

@@ -2,7 +2,7 @@
#
# This file is part of Nominatim. (https://nominatim.org)
#
# Copyright (C) 2025 by the Nominatim developer community.
# Copyright (C) 2024 by the Nominatim developer community.
# For a full list of authors see the git log.
"""
Hard-coded information about tag categories.
@@ -20,9 +20,7 @@ def get_label_tag(category: Tuple[str, str], extratags: Optional[Mapping[str, st
rank: int, country: Optional[str]) -> str:
""" Create a label tag for the given place that can be used as an XML name.
"""
if category in (('place', 'postcode'), ('boundary', 'postal_code')):
label = 'postcode'
elif rank < 26 and extratags and 'place' in extratags:
if rank < 26 and extratags and 'place' in extratags:
label = extratags['place']
elif rank < 26 and extratags and 'linked_place' in extratags:
label = extratags['linked_place']
@@ -30,6 +28,8 @@ def get_label_tag(category: Tuple[str, str], extratags: Optional[Mapping[str, st
label = ADMIN_LABELS.get((country or '', rank // 2))\
or ADMIN_LABELS.get(('', rank // 2))\
or 'Administrative'
elif category[1] == 'postal_code':
label = 'postcode'
elif rank < 26:
label = category[1] if category[1] != 'yes' else category[0]
elif rank < 28:

View File

@@ -132,7 +132,7 @@ def _add_parent_rows_grouped(writer: JsonWriter, rows: AddressLines,
@dispatch.format_func(DetailedResult, 'json')
def _format_details_json(result: DetailedResult, options: Mapping[str, Any]) -> str:
locales = options.get('locales') or Locales()
locales = options.get('locales', Locales())
geom = result.geometry.get('geojson')
centroid = result.centroid.to_geojson()

View File

@@ -2,7 +2,7 @@
#
# This file is part of Nominatim. (https://nominatim.org)
#
# Copyright (C) 2025 by the Nominatim developer community.
# Copyright (C) 2024 by the Nominatim developer community.
# For a full list of authors see the git log.
"""
Helper function for parsing parameters and and outputting data
@@ -12,7 +12,7 @@ from typing import Tuple, Optional, Any, Dict, Iterable
from itertools import chain
import re
from ..results import SearchResults, SourceTable
from ..results import SearchResult, SearchResults, SourceTable
from ..types import SearchDetails, GeometryFormat
@@ -106,6 +106,10 @@ def deduplicate_results(results: SearchResults, max_results: int) -> SearchResul
classification_done = set()
deduped = SearchResults()
for result in results:
if result.source_table == SourceTable.POSTCODE:
assert result.names and 'ref' in result.names
if any(_is_postcode_relation_for(r, result.names['ref']) for r in results):
continue
if result.source_table == SourceTable.PLACEX:
classification = (result.osm_object[0] if result.osm_object else None,
result.category,
@@ -124,6 +128,15 @@ def deduplicate_results(results: SearchResults, max_results: int) -> SearchResul
return deduped
def _is_postcode_relation_for(result: SearchResult, postcode: str) -> bool:
return result.source_table == SourceTable.PLACEX \
and result.osm_object is not None \
and result.osm_object[0] == 'R' \
and result.category == ('boundary', 'postal_code') \
and result.names is not None \
and result.names.get('ref') == postcode
def _deg(axis: str) -> str:
return f"(?P<{axis}_deg>\\d+\\.\\d+)°?"

View File

@@ -2,7 +2,7 @@
#
# This file is part of Nominatim. (https://nominatim.org)
#
# Copyright (C) 2026 by the Nominatim developer community.
# Copyright (C) 2025 by the Nominatim developer community.
# For a full list of authors see the git log.
"""
Generic part of the server implementation of the v1 API.
@@ -174,8 +174,7 @@ async def details_endpoint(api: NominatimAPIAsync, params: ASGIAdaptor) -> Any:
if result is None:
params.raise_error('No place with that OSM ID found.', status=404)
locales = Locales.from_accept_languages(get_accepted_languages(params),
params.config().OUTPUT_NAMES)
locales = Locales.from_accept_languages(get_accepted_languages(params))
locales.localize_results([result])
output = params.formatting().format_result(
@@ -200,7 +199,6 @@ async def reverse_endpoint(api: NominatimAPIAsync, params: ASGIAdaptor) -> Any:
details['max_rank'] = helpers.zoom_to_rank(params.get_int('zoom', 18))
details['layers'] = get_layers(params)
details['query_stats'] = params.query_stats()
details['entrances'] = params.get_bool('entrances', False)
result = await api.reverse(coord, **details)
@@ -217,8 +215,8 @@ async def reverse_endpoint(api: NominatimAPIAsync, params: ASGIAdaptor) -> Any:
query = ''
if result:
Locales.from_accept_languages(get_accepted_languages(params),
params.config().OUTPUT_NAMES).localize_results([result])
Locales.from_accept_languages(get_accepted_languages(params)).localize_results(
[result])
fmt_options = {'query': query,
'extratags': params.get_bool('extratags', False),
@@ -239,7 +237,6 @@ async def lookup_endpoint(api: NominatimAPIAsync, params: ASGIAdaptor) -> Any:
debug = setup_debugging(params)
details = parse_geometry_details(params, fmt)
details['query_stats'] = params.query_stats()
details['entrances'] = params.get_bool('entrances', False)
places = []
for oid in (params.get('osm_ids') or '').split(','):
@@ -258,8 +255,7 @@ async def lookup_endpoint(api: NominatimAPIAsync, params: ASGIAdaptor) -> Any:
if debug:
return build_response(params, loglib.get_and_disable(), num_results=len(results))
Locales.from_accept_languages(get_accepted_languages(params),
params.config().OUTPUT_NAMES).localize_results(results)
Locales.from_accept_languages(get_accepted_languages(params)).localize_results(results)
fmt_options = {'extratags': params.get_bool('extratags', False),
'namedetails': params.get_bool('namedetails', False),
@@ -352,8 +348,7 @@ async def search_endpoint(api: NominatimAPIAsync, params: ASGIAdaptor) -> Any:
except UsageError as err:
params.raise_error(str(err))
Locales.from_accept_languages(get_accepted_languages(params),
params.config().OUTPUT_NAMES).localize_results(results)
Locales.from_accept_languages(get_accepted_languages(params)).localize_results(results)
if details['dedupe'] and len(results) > 1:
results = helpers.deduplicate_results(results, max_results)

View File

@@ -65,14 +65,14 @@ class UpdateAddData:
def run(self, args: NominatimArgs) -> int:
from ..tools import add_osm_data
if args.tiger_data:
return asyncio.run(self._add_tiger_data(args))
with connect(args.config.get_libpq_dsn()) as conn:
if is_frozen(conn):
print('Database is marked frozen. New data can\'t be added.')
return 1
if args.tiger_data:
return asyncio.run(self._add_tiger_data(args))
osm2pgsql_params = args.osm2pgsql_options(default_cache=1000, default_threads=1)
if args.file or args.diff:
return add_osm_data.add_data_from_file(args.config.get_libpq_dsn(),

View File

@@ -19,7 +19,6 @@ import nominatim_api as napi
from nominatim_api.v1.helpers import zoom_to_rank, deduplicate_results
from nominatim_api.server.content_types import CONTENT_JSON
import nominatim_api.logging as loglib
from ..config import Configuration
from ..errors import UsageError
from .args import NominatimArgs
@@ -92,19 +91,18 @@ def _get_geometry_output(args: NominatimArgs) -> napi.GeometryFormat:
raise UsageError(f"Unknown polygon output format '{args.polygon_output}'.") from exp
def _get_locales(args: NominatimArgs, config: Configuration) -> napi.Locales:
def _get_locales(args: NominatimArgs, default: Optional[str]) -> napi.Locales:
""" Get the locales from the language parameter.
"""
language = args.lang or config.DEFAULT_LANGUAGE
output_names = config.OUTPUT_NAMES
if language:
return napi.Locales.from_accept_languages(language, output_names)
if args.lang:
return napi.Locales.from_accept_languages(args.lang)
if default:
return napi.Locales.from_accept_languages(default)
return napi.Locales()
def _get_layers(args: NominatimArgs, default: Optional[napi.DataLayer]) -> Optional[napi.DataLayer]:
def _get_layers(args: NominatimArgs, default: napi.DataLayer) -> Optional[napi.DataLayer]:
""" Get the list of selected layers as a DataLayer enum.
"""
if not args.layers:
@@ -136,7 +134,7 @@ def _print_output(formatter: napi.FormatDispatcher, result: Any,
json.dump(json.loads(output), sys.stdout, indent=4, ensure_ascii=False)
except json.decoder.JSONDecodeError as err:
# Catch the error here, so that data can be debugged,
# when people are developing custom result formatters.
# when people are developping custom result formatters.
LOG.fatal("Parsing json failed: %s\nUnformatted output:\n%s", err, output)
else:
sys.stdout.write(output)
@@ -173,10 +171,6 @@ class APISearch:
help='Preferred area to find search results')
group.add_argument('--bounded', action='store_true',
help='Strictly restrict results to viewbox area')
group.add_argument('--layer', metavar='LAYER',
choices=[n.name.lower() for n in napi.DataLayer if n.name],
action='append', required=False, dest='layers',
help='Restrict results to one or more layers (may be repeated)')
group.add_argument('--no-dedupe', action='store_false', dest='dedupe',
help='Do not remove duplicates from the result list')
_add_list_format(parser)
@@ -193,8 +187,6 @@ class APISearch:
raise UsageError(f"Unsupported format '{args.format}'. "
'Use --list-formats to see supported formats.')
layers = _get_layers(args, None)
try:
with napi.NominatimAPI(args.project_dir) as api:
params: Dict[str, Any] = {'max_results': args.limit + min(args.limit, 10),
@@ -205,7 +197,6 @@ class APISearch:
'excluded': args.exclude_place_ids,
'viewbox': args.viewbox,
'bounded_viewbox': args.bounded,
'layers': layers,
'entrances': args.entrances,
}
@@ -223,7 +214,7 @@ class APISearch:
except napi.UsageError as ex:
raise UsageError(ex) from ex
locales = _get_locales(args, api.config)
locales = _get_locales(args, api.config.DEFAULT_LANGUAGE)
locales.localize_results(results)
if args.dedupe and len(results) > 1:
@@ -262,7 +253,7 @@ class APIReverse:
group.add_argument('--layer', metavar='LAYER',
choices=[n.name.lower() for n in napi.DataLayer if n.name],
action='append', required=False, dest='layers',
help='Restrict results to one or more layers (may be repeated)')
help='OSM id to lookup in format <NRW><id> (may be repeated)')
_add_api_output_arguments(parser)
_add_list_format(parser)
@@ -296,7 +287,7 @@ class APIReverse:
raise UsageError(ex) from ex
if result is not None:
locales = _get_locales(args, api.config)
locales = _get_locales(args, api.config.DEFAULT_LANGUAGE)
locales.localize_results([result])
if args.format == 'debug':
@@ -361,7 +352,7 @@ class APILookup:
except napi.UsageError as ex:
raise UsageError(ex) from ex
locales = _get_locales(args, api.config)
locales = _get_locales(args, api.config.DEFAULT_LANGUAGE)
locales.localize_results(results)
if args.format == 'debug':
@@ -461,7 +452,7 @@ class APIDetails:
raise UsageError(ex) from ex
if result is not None:
locales = _get_locales(args, api.config)
locales = _get_locales(args, api.config.DEFAULT_LANGUAGE)
locales.localize_results([result])
if args.format == 'debug':

View File

@@ -119,7 +119,6 @@ class NominatimArgs:
enable_debug_statements: bool
data_object: Sequence[Tuple[str, int]]
data_area: Sequence[Tuple[str, int]]
ro_access: bool
# Arguments to 'replication'
init: bool

View File

@@ -154,7 +154,7 @@ async def dump_results(conn: napi.SearchConnection,
await add_result_details(conn, results,
LookupDetails(address_details=True))
locale = napi.Locales([lang] if lang else None, conn.config.OUTPUT_NAMES)
locale = napi.Locales([lang] if lang else None)
locale.localize_results(results)
for result in results:

View File

@@ -65,8 +65,6 @@ class UpdateRefresh:
help='Update secondary importance raster data')
group.add_argument('--importance', action='store_true',
help='Recompute place importances (expensive!)')
group.add_argument('--ro-access', action='store_true',
help='Grant read-only access to web user for all tables')
group.add_argument('--website', action='store_true',
help='DEPRECATED. This function has no function anymore'
' and will be removed in a future version.')
@@ -161,11 +159,6 @@ class UpdateRefresh:
LOG.error('WARNING: Website setup is no longer required. '
'This function will be removed in future version of Nominatim.')
if args.ro_access:
from ..tools import admin
LOG.warning('Grant read-only access to web user')
admin.grant_ro_access(args.config.get_libpq_dsn(), args.config)
if args.data_object or args.data_area:
with connect(args.config.get_libpq_dsn()) as conn:
for obj in args.data_object or []:

View File

@@ -23,7 +23,6 @@ from ..tokenizer.base import AbstractTokenizer
from ..version import NOMINATIM_VERSION
from .args import NominatimArgs
import time
LOG = logging.getLogger()
@@ -87,8 +86,6 @@ class SetupAll:
from ..tools import database_import, postcodes, freeze
from ..indexer.indexer import Indexer
start_time = time.time()
num_threads = args.threads or psutil.cpu_count() or 1
country_info.setup_country_config(args.config)
@@ -141,10 +138,6 @@ class SetupAll:
LOG.warning('Recompute word counts')
tokenizer.update_statistics(args.config, threads=num_threads)
end_time = time.time()
elapsed = end_time - start_time
LOG.warning(f'Import completed successfully in {elapsed:.2f} seconds.')
self._finalize_database(args.config.get_libpq_dsn(), args.offline)
return 0

View File

@@ -197,7 +197,7 @@ class Configuration:
if dsn.startswith('pgsql:'):
return dict((p.split('=', 1) for p in dsn[6:].split(';')))
return conninfo_to_dict(dsn) # type: ignore
return conninfo_to_dict(dsn)
def get_import_style_file(self) -> Path:
""" Return the import style file as a path object. Translates the

View File

@@ -2,7 +2,7 @@
#
# This file is part of Nominatim. (https://nominatim.org)
#
# Copyright (C) 2025 by the Nominatim developer community.
# Copyright (C) 2024 by the Nominatim developer community.
# For a full list of authors see the git log.
"""
Functions for formatting postcodes according to their country-specific
@@ -29,9 +29,6 @@ class CountryPostcodeMatcher:
self.norm_pattern = re.compile(f'\\s*(?:{country_code.upper()}[ -]?)?({pc_pattern})\\s*')
self.pattern = re.compile(pc_pattern)
# We want to exclude 0000, 00-000, 000 00 etc
self.zero_pattern = re.compile(r'^[0\- ]+$')
self.output = config.get('output', r'\g<0>')
def match(self, postcode: str) -> Optional[Match[str]]:
@@ -43,10 +40,7 @@ class CountryPostcodeMatcher:
normalized = self.norm_pattern.fullmatch(postcode.upper())
if normalized:
match = self.pattern.fullmatch(normalized.group(1))
if match and self.zero_pattern.match(match.string):
return None
return match
return self.pattern.fullmatch(normalized.group(1))
return None
@@ -67,15 +61,12 @@ class PostcodeFormatter:
self.country_without_postcode: Set[Optional[str]] = {None}
self.country_matcher = {}
self.default_matcher = CountryPostcodeMatcher('', {'pattern': '.*'})
self.postcode_extent: dict[Optional[str], int] = {}
for ccode, prop in country_info.iterate('postcode'):
if prop is False:
self.country_without_postcode.add(ccode)
elif isinstance(prop, dict):
self.country_matcher[ccode] = CountryPostcodeMatcher(ccode, prop)
if 'extent' in prop:
self.postcode_extent[ccode] = int(prop['extent'])
else:
raise UsageError(f"Invalid entry 'postcode' for country '{ccode}'")
@@ -116,9 +107,3 @@ class PostcodeFormatter:
`match()`
"""
return self.country_matcher.get(country_code, self.default_matcher).normalize(match)
def get_postcode_extent(self, country_code: Optional[str]) -> int:
""" Return the extent (in m) to use for the given country. If no
specific extent is set, then the default of 5km will be returned.
"""
return self.postcode_extent.get(country_code, 5000)

View File

@@ -2,7 +2,7 @@
#
# This file is part of Nominatim. (https://nominatim.org)
#
# Copyright (C) 2026 by the Nominatim developer community.
# Copyright (C) 2024 by the Nominatim developer community.
# For a full list of authors see the git log.
"""
A connection pool that executes incoming queries in parallel.
@@ -27,28 +27,20 @@ class QueryPool:
The results of the queries is discarded.
"""
def __init__(self, dsn: str, pool_size: int = 1, **conn_args: Any) -> None:
self.is_cancelled = False
self.wait_time = 0.0
self.query_queue: 'asyncio.Queue[QueueItem]' = asyncio.Queue(maxsize=2 * pool_size)
self.pool = [asyncio.create_task(self._worker_loop_cancellable(dsn, **conn_args))
self.pool = [asyncio.create_task(self._worker_loop(dsn, **conn_args))
for _ in range(pool_size)]
async def put_query(self, query: psycopg.abc.Query, params: Any) -> None:
""" Schedule a query for execution.
"""
if self.is_cancelled:
await self.finish()
return
tstart = time.time()
await self.query_queue.put((query, params))
self.wait_time += time.time() - tstart
await asyncio.sleep(0)
if self.is_cancelled:
await self.finish()
async def finish(self) -> None:
""" Wait for all queries to finish and close the pool.
"""
@@ -64,25 +56,6 @@ class QueryPool:
if excp is not None:
raise excp
def clear_queue(self) -> None:
""" Drop all items silently that might still be queued.
"""
try:
while True:
self.query_queue.get_nowait()
except asyncio.QueueEmpty:
pass # expected
async def _worker_loop_cancellable(self, dsn: str, **conn_args: Any) -> None:
try:
await self._worker_loop(dsn, **conn_args)
except Exception as e:
# Make sure the exception is forwarded to the main function
self.is_cancelled = True
# clear the queue here to ensure that any put() that may be blocked returns
self.clear_queue()
raise e
async def _worker_loop(self, dsn: str, **conn_args: Any) -> None:
conn_args['autocommit'] = True
aconn = await psycopg.AsyncConnection.connect(dsn, **conn_args)

View File

@@ -2,7 +2,7 @@
#
# This file is part of Nominatim. (https://nominatim.org)
#
# Copyright (C) 2025 by the Nominatim developer community.
# Copyright (C) 2024 by the Nominatim developer community.
# For a full list of authors see the git log.
"""
Main work horse for indexing (computing addresses) the database.
@@ -154,7 +154,7 @@ class Indexer:
return total
async def index_postcodes(self) -> int:
"""Index the entries of the location_postcodes table.
"""Index the entries of the location_postcode table.
"""
LOG.warning("Starting indexing postcodes using %s threads", self.num_threads)
@@ -177,7 +177,7 @@ class Indexer:
`total_tuples` may contain the total number of rows to process.
When not supplied, the value will be computed using the
appropriate runner function.
approriate runner function.
"""
LOG.warning("Starting %s (using batch size %s)", runner.name(), batch)

View File

@@ -2,7 +2,7 @@
#
# This file is part of Nominatim. (https://nominatim.org)
#
# Copyright (C) 2025 by the Nominatim developer community.
# Copyright (C) 2024 by the Nominatim developer community.
# For a full list of authors see the git log.
"""
Mix-ins that provide the actual commands for the indexer for various indexing
@@ -143,22 +143,22 @@ class InterpolationRunner:
class PostcodeRunner(Runner):
""" Provides the SQL commands for indexing the location_postcodes table.
""" Provides the SQL commands for indexing the location_postcode table.
"""
def name(self) -> str:
return "postcodes (location_postcodes)"
return "postcodes (location_postcode)"
def sql_count_objects(self) -> Query:
return 'SELECT count(*) FROM location_postcodes WHERE indexed_status > 0'
return 'SELECT count(*) FROM location_postcode WHERE indexed_status > 0'
def sql_get_objects(self) -> Query:
return """SELECT place_id FROM location_postcodes
return """SELECT place_id FROM location_postcode
WHERE indexed_status > 0
ORDER BY country_code, postcode"""
def index_places_query(self, batch_size: int) -> Query:
return pysql.SQL("""UPDATE location_postcodes SET indexed_status = 0
return pysql.SQL("""UPDATE location_postcode SET indexed_status = 0
WHERE place_id IN ({})""")\
.format(pysql.SQL(',').join((pysql.Placeholder() for _ in range(batch_size))))

View File

@@ -71,7 +71,7 @@ class AbstractAnalyzer(ABC):
@abstractmethod
def update_postcodes_from_db(self) -> None:
""" Update the tokenizer's postcode tokens from the current content
of the `location_postcodes` table.
of the `location_postcode` table.
"""
@abstractmethod

View File

@@ -475,23 +475,20 @@ class ICUNameAnalyzer(AbstractAnalyzer):
assert self.conn is not None
word_tokens = set()
for name in names:
norm_name = self._normalized(name.name)
token_name = self._search_normalized(name.name)
if norm_name and token_name:
word_tokens.add((token_name, norm_name))
norm_name = self._search_normalized(name.name)
if norm_name:
word_tokens.add(norm_name)
with self.conn.cursor() as cur:
# Get existing names
cur.execute("""SELECT word_token,
word as lookup,
coalesce(info ? 'internal', false) as is_internal
cur.execute("""SELECT word_token, coalesce(info ? 'internal', false) as is_internal
FROM word
WHERE type = 'C' and info->>'cc' = %s""",
WHERE type = 'C' and word = %s""",
(country_code, ))
# internal/external names
existing_tokens: Dict[bool, Set[Tuple[str, str]]] = {True: set(), False: set()}
existing_tokens: Dict[bool, Set[str]] = {True: set(), False: set()}
for word in cur:
existing_tokens[word[2]].add((word[0], word[1]))
existing_tokens[word[1]].add(word[0])
# Delete names that no longer exist.
gone_tokens = existing_tokens[internal] - word_tokens
@@ -499,10 +496,10 @@ class ICUNameAnalyzer(AbstractAnalyzer):
gone_tokens.update(existing_tokens[False] & word_tokens)
if gone_tokens:
cur.execute("""DELETE FROM word
USING jsonb_array_elements(%s) as data
WHERE type = 'C' and info->>'cc' = %s
and word_token = data->>0 and word = data->>1""",
(Jsonb(list(gone_tokens)), country_code))
USING unnest(%s::text[]) as token
WHERE type = 'C' and word = %s
and word_token = token""",
(list(gone_tokens), country_code))
# Only add those names that are not yet in the list.
new_tokens = word_tokens - existing_tokens[True]
@@ -511,17 +508,15 @@ class ICUNameAnalyzer(AbstractAnalyzer):
if new_tokens:
if internal:
sql = """INSERT INTO word (word_token, type, word, info)
(SELECT data->>0, 'C', data->>1,
jsonb_build_object('internal', 'yes', 'cc', %s::text)
FROM jsonb_array_elements(%s) as data)
(SELECT token, 'C', %s, '{"internal": "yes"}'
FROM unnest(%s::text[]) as token)
"""
else:
sql = """INSERT INTO word (word_token, type, word, info)
(SELECT data->>0, 'C', data->>1,
jsonb_build_object('cc', %s::text)
FROM jsonb_array_elements(%s) as data)
sql = """INSERT INTO word (word_token, type, word)
(SELECT token, 'C', %s
FROM unnest(%s::text[]) as token)
"""
cur.execute(sql, (country_code, Jsonb(list(new_tokens))))
cur.execute(sql, (country_code, list(new_tokens)))
def process_place(self, place: PlaceInfo) -> Mapping[str, Any]:
""" Determine tokenizer information about the given place.

View File

@@ -16,7 +16,6 @@ from psycopg.types.json import Json
from ..typing import DictCursorResult
from ..config import Configuration
from ..db.connection import connect, Cursor, register_hstore
from ..db.sql_preprocessor import SQLPreprocessor
from ..errors import UsageError
from ..tokenizer import factory as tokenizer_factory
from ..data.place_info import PlaceInfo
@@ -106,12 +105,3 @@ def clean_deleted_relations(config: Configuration, age: str) -> None:
except psycopg.DataError as exc:
raise UsageError('Invalid PostgreSQL time interval format') from exc
conn.commit()
def grant_ro_access(dsn: str, config: Configuration) -> None:
""" Grant read-only access to the web user for all Nominatim tables.
This can be used to grant access to a different user after import.
"""
with connect(dsn) as conn:
sql = SQLPreprocessor(conn, config)
sql.run_sql_file(conn, 'grants.sql')

View File

@@ -113,8 +113,8 @@ def _get_indexes(conn: Connection) -> List[str]:
'idx_placex_geometry_placenode',
'idx_osmline_parent_place_id',
'idx_osmline_parent_osm_id',
'idx_location_postcodes_id',
'idx_location_postcodes_postcode'
'idx_postcode_id',
'idx_postcode_postcode'
]
# These won't exist if --reverse-only import was used

View File

@@ -2,7 +2,7 @@
#
# This file is part of Nominatim. (https://nominatim.org)
#
# Copyright (C) 2025 by the Nominatim developer community.
# Copyright (C) 2024 by the Nominatim developer community.
# For a full list of authors see the git log.
"""
Functions for setting up and importing a new Nominatim database.
@@ -157,8 +157,6 @@ def create_tables(conn: Connection, config: Configuration, reverse_only: bool =
sql.run_sql_file(conn, 'tables.sql')
sql.run_sql_file(conn, 'grants.sql')
def create_table_triggers(conn: Connection, config: Configuration) -> None:
""" Create the triggers for the tables. The trigger functions must already
@@ -185,7 +183,7 @@ def truncate_data_tables(conn: Connection) -> None:
cur.execute('TRUNCATE location_area_country')
cur.execute('TRUNCATE location_property_tiger')
cur.execute('TRUNCATE location_property_osmline')
cur.execute('TRUNCATE location_postcodes')
cur.execute('TRUNCATE location_postcode')
if table_exists(conn, 'search_name'):
cur.execute('TRUNCATE search_name')
cur.execute('DROP SEQUENCE IF EXISTS seq_place')
@@ -227,7 +225,7 @@ async def load_data(dsn: str, threads: int) -> None:
total=pysql.Literal(placex_threads),
mod=pysql.Literal(imod)), None)
# Interpolations need to be copied separately
# Interpolations need to be copied seperately
await pool.put_query("""
INSERT INTO location_property_osmline (osm_id, address, linegeo)
SELECT osm_id, address, geometry FROM place

View File

@@ -2,7 +2,7 @@
#
# This file is part of Nominatim. (https://nominatim.org)
#
# Copyright (C) 2025 by the Nominatim developer community.
# Copyright (C) 2024 by the Nominatim developer community.
# For a full list of authors see the git log.
"""
Functions for removing unnecessary data from the database.
@@ -18,11 +18,10 @@ UPDATE_TABLES = [
'address_levels',
'gb_postcode',
'import_osmosis_log',
'import_polygon_%',
'location_area%',
'location_road%',
'place',
'place_entrance',
'place_postcode',
'planet_osm_%',
'search_name_%',
'us_postcode',

View File

@@ -2,7 +2,7 @@
#
# This file is part of Nominatim. (https://nominatim.org)
#
# Copyright (C) 2025 by the Nominatim developer community.
# Copyright (C) 2024 by the Nominatim developer community.
# For a full list of authors see the git log.
"""
Functions for database migration to newer software versions.
@@ -18,8 +18,6 @@ from ..db.connection import connect, Connection, \
from ..db.sql_preprocessor import SQLPreprocessor
from ..version import NominatimVersion, NOMINATIM_VERSION, parse_version
from ..tokenizer import factory as tokenizer_factory
from ..data.country_info import create_country_names, setup_country_config
from .freeze import is_frozen
from . import refresh
LOG = logging.getLogger()
@@ -29,7 +27,7 @@ _MIGRATION_FUNCTIONS: List[Tuple[NominatimVersion, Callable[..., None]]] = []
def migrate(config: Configuration, paths: Any) -> int:
""" Check for the current database version and execute migrations,
if necessary.
if necesssary.
"""
with connect(config.get_libpq_dsn()) as conn:
register_hstore(conn)
@@ -143,7 +141,7 @@ def create_placex_entrance_table(conn: Connection, config: Configuration, **_: A
@_migration(5, 1, 99, 1)
def create_place_entrance_table(conn: Connection, config: Configuration, **_: Any) -> None:
""" Add the place_entrance table to store incoming entrance nodes
""" Add the place_entrance table to store incomming entrance nodes
"""
if not table_exists(conn, 'place_entrance'):
with conn.cursor() as cur:
@@ -158,195 +156,3 @@ def create_place_entrance_table(conn: Connection, config: Configuration, **_: An
CREATE UNIQUE INDEX place_entrance_osm_id_idx ON place_entrance
USING BTREE (osm_id);
""")
@_migration(5, 2, 99, 0)
def convert_country_tokens(conn: Connection, config: Configuration, **_: Any) -> None:
""" Convert country word tokens
Country tokens now save the country in the info field instead of the
word. This migration removes all country tokens from the word table
and reimports the default country name. This means that custom names
are lost. If you need them back, invalidate the OSM objects containing
the names by setting indexed_status to 2 and then reindex the database.
"""
tokenizer = tokenizer_factory.get_tokenizer_for_db(config)
# There is only one tokenizer at the time of migration, so we make
# some assumptions here about the structure of the database. This will
# fail if somebody has written a custom tokenizer.
with conn.cursor() as cur:
cur.execute("DELETE FROM word WHERE type = 'C'")
conn.commit()
setup_country_config(config)
create_country_names(conn, tokenizer, config.get_str_list('LANGUAGES'))
@_migration(5, 2, 99, 1)
def create_place_postcode_table(conn: Connection, config: Configuration, **_: Any) -> None:
""" Restructure postcode tables
"""
sqlp = SQLPreprocessor(conn, config)
mutable = not is_frozen(conn)
has_place_table = table_exists(conn, 'place_postcode')
has_postcode_table = table_exists(conn, 'location_postcodes')
if mutable and not has_place_table:
with conn.cursor() as cur:
cur.execute(
"""
CREATE TABLE place_postcode (
osm_type CHAR(1) NOT NULL,
osm_id BIGINT NOT NULL,
postcode TEXT NOT NULL,
country_code TEXT,
centroid GEOMETRY(Point, 4326) NOT NULL,
geometry GEOMETRY(Geometry, 4326)
)
""")
# Move postcode points into the new table
cur.execute("ALTER TABLE place DISABLE TRIGGER ALL")
cur.execute(
"""
WITH deleted AS (
DELETE FROM place
WHERE (class = 'place' AND type = 'postcode')
OR (osm_type = 'R'
AND class = 'boundary' AND type = 'postal_code')
RETURNING osm_type, osm_id, address->'postcode' as postcode,
ST_Centroid(geometry) as centroid,
(CASE WHEN class = 'place' THEN NULL ELSE geometry END) as geometry)
INSERT INTO place_postcode (osm_type, osm_id, postcode, centroid, geometry)
(SELECT * FROM deleted
WHERE deleted.postcode is not NULL AND deleted.centroid is not NULL)
""")
cur.execute(
"""
CREATE INDEX place_postcode_osm_id_idx ON place_postcode
USING BTREE (osm_type, osm_id)
""")
cur.execute(
"""
CREATE INDEX place_postcode_postcode_idx ON place_postcode
USING BTREE (postcode)
""")
cur.execute("ALTER TABLE place ENABLE TRIGGER ALL")
if not has_postcode_table:
sqlp.run_sql_file(conn, 'functions/postcode_triggers.sql')
with conn.cursor() as cur:
# create a new location_postcode table which will replace the
# old one atomically in the end
cur.execute(
"""
CREATE TABLE location_postcodes (
place_id BIGINT,
osm_id BIGINT,
rank_search SMALLINT,
parent_place_id BIGINT,
indexed_status SMALLINT,
indexed_date TIMESTAMP,
country_code VARCHAR(2),
postcode TEXT,
centroid Geometry(Point, 4326),
geometry Geometry(Geometry, 4326) NOT NULL
)
""")
sqlp.run_string(conn,
'GRANT SELECT ON location_postcodes TO "{{config.DATABASE_WEBUSER}}"')
# remove postcodes from the various auxiliary tables
cur.execute(
"""
DELETE FROM place_addressline
WHERE address_place_id = ANY(
SELECT place_id FROM placex
WHERE osm_type = 'R'
AND class = 'boundary' AND type = 'postal_code')
""")
if mutable:
cur.execute(
"""
SELECT deleteLocationArea(partition, place_id, rank_search),
deleteSearchName(partition, place_id)
FROM placex
WHERE osm_type = 'R' AND class = 'boundary' AND type = 'postal_code'
""")
if table_exists(conn, 'search_name'):
cur.execute(
"""
DELETE FROM search_name
WHERE place_id = ANY(
SELECT place_id FROM placex
WHERE osm_type = 'R'
AND class = 'boundary' AND type = 'postal_code')
""")
# move postcode areas from placex to location_postcodes
# avoiding automatic invalidation
cur.execute("ALTER TABLE placex DISABLE TRIGGER ALL")
cur.execute(
"""
WITH deleted AS (
DELETE FROM placex
WHERE osm_type = 'R'
AND class = 'boundary' AND type = 'postal_code'
RETURNING place_id, osm_id, rank_search, parent_place_id,
indexed_status, indexed_date,
country_code, postcode, centroid, geometry)
INSERT INTO location_postcodes (SELECT * from deleted)
""")
cur.execute("ALTER TABLE placex ENABLE TRIGGER ALL")
# remove any old postcode centroid that would overlap with areas
cur.execute(
"""
DELETE FROM location_postcode o USING location_postcodes n
WHERE o.country_code = n.country_code
AND o.postcode = n.postcode
""")
# copy over old postcodes
cur.execute(
"""
INSERT INTO location_postcodes
(SELECT place_id, NULL, rank_search, parent_place_id,
indexed_status, indexed_date, country_code,
postcode, geometry,
ST_Expand(geometry, 0.05)
FROM location_postcode)
""")
# add indexes and triggers
cur.execute("""CREATE INDEX idx_location_postcodes_geometry
ON location_postcodes USING GIST(geometry)""")
cur.execute("""CREATE INDEX idx_location_postcodes_id
ON location_postcodes USING BTREE(place_id)""")
cur.execute("""CREATE INDEX idx_location_postcodes_osmid
ON location_postcodes USING BTREE(osm_id)""")
cur.execute("""CREATE INDEX idx_location_postcodes_postcode
ON location_postcodes USING BTREE(postcode, country_code)""")
cur.execute("""CREATE INDEX idx_location_postcodes_parent_place_id
ON location_postcodes USING BTREE(parent_place_id)""")
cur.execute("""CREATE TRIGGER location_postcodes_before_update
BEFORE UPDATE ON location_postcodes
FOR EACH ROW EXECUTE PROCEDURE postcodes_update()""")
cur.execute("""CREATE TRIGGER location_postcodes_before_delete
BEFORE DELETE ON location_postcodes
FOR EACH ROW EXECUTE PROCEDURE postcodes_delete()""")
cur.execute("""CREATE TRIGGER location_postcodes_before_insert
BEFORE INSERT ON location_postcodes
FOR EACH ROW EXECUTE PROCEDURE postcodes_insert()""")
sqlp.run_string(
conn,
"""
CREATE INDEX IF NOT EXISTS idx_placex_geometry_reverse_lookupPolygon_nopostcode
ON placex USING gist (geometry) {{db.tablespace.search_index}}
WHERE St_GeometryType(geometry) in ('ST_Polygon', 'ST_MultiPolygon')
AND rank_address between 4 and 25
AND name is not null AND linked_place_id is null;
CREATE INDEX IF NOT EXISTS idx_placex_geometry_reverse_lookupPlaceNode_nopostcode
ON placex USING gist (ST_Buffer(geometry, reverse_place_diameter(rank_search)))
{{db.tablespace.search_index}}
WHERE rank_address between 4 and 25
AND name is not null AND linked_place_id is null AND osm_type = 'N';
CREATE INDEX idx_placex_geometry_placenode_nopostcode ON placex
USING SPGIST (geometry) {{db.tablespace.address_index}}
WHERE osm_type = 'N' and rank_search < 26 and class = 'place';
ANALYSE;
""")

View File

@@ -8,7 +8,7 @@
Functions for importing, updating and otherwise maintaining the table
of artificial postcode centroids.
"""
from typing import Optional, Tuple, Dict, TextIO
from typing import Optional, Tuple, Dict, List, TextIO
from collections import defaultdict
from pathlib import Path
import csv
@@ -38,26 +38,13 @@ def _to_float(numstr: str, max_value: float) -> float:
return num
def _extent_to_rank(extent: int) -> int:
""" Guess a suitable search rank from the extent of a postcode.
"""
if extent <= 100:
return 25
if extent <= 3000:
return 23
return 21
class _PostcodeCollector:
""" Collector for postcodes of a single country.
"""
def __init__(self, country: str, matcher: Optional[CountryPostcodeMatcher],
default_extent: int, exclude: set[str] = set()):
def __init__(self, country: str, matcher: Optional[CountryPostcodeMatcher]):
self.country = country
self.matcher = matcher
self.extent = default_extent
self.exclude = exclude
self.collected: Dict[str, PointsCentroid] = defaultdict(PointsCentroid)
self.normalization_cache: Optional[Tuple[str, Optional[str]]] = None
@@ -74,7 +61,7 @@ class _PostcodeCollector:
normalized = self.matcher.normalize(match) if match else None
self.normalization_cache = (postcode, normalized)
if normalized and normalized not in self.exclude:
if normalized:
self.collected[normalized] += (x, y)
def commit(self, conn: Connection, analyzer: AbstractAnalyzer,
@@ -86,38 +73,61 @@ class _PostcodeCollector:
"""
if project_dir is not None:
self._update_from_external(analyzer, project_dir)
to_add, to_delete, to_update = self._compute_changes(conn)
with conn.cursor() as cur:
cur.execute("""SELECT postcode FROM location_postcodes
WHERE country_code = %s AND osm_id is null""",
(self.country, ))
to_delete = [row[0] for row in cur if row[0] not in self.collected]
to_add = [dict(zip(('pc', 'x', 'y'), (k, *v.centroid())))
for k, v in self.collected.items()]
self.collected = defaultdict(PointsCentroid)
LOG.info("Processing country '%s' (%s added, %s deleted).",
self.country, len(to_add), len(to_delete))
LOG.info("Processing country '%s' (%s added, %s deleted, %s updated).",
self.country, len(to_add), len(to_delete), len(to_update))
with conn.cursor() as cur:
if to_add:
cur.executemany(pysql.SQL(
"""INSERT INTO location_postcodes
(country_code, rank_search, postcode, centroid, geometry)
VALUES ({}, {}, %(pc)s,
ST_SetSRID(ST_MakePoint(%(x)s, %(y)s), 4326),
expand_by_meters(ST_SetSRID(ST_MakePoint(%(x)s, %(y)s), 4326), {}))
""").format(pysql.Literal(self.country),
pysql.Literal(_extent_to_rank(self.extent)),
pysql.Literal(self.extent)),
"""INSERT INTO location_postcode
(place_id, indexed_status, country_code,
postcode, geometry)
VALUES (nextval('seq_place'), 1, {}, %s,
ST_SetSRID(ST_MakePoint(%s, %s), 4326))
""").format(pysql.Literal(self.country)),
to_add)
if to_delete:
cur.execute("""DELETE FROM location_postcodes
cur.execute("""DELETE FROM location_postcode
WHERE country_code = %s and postcode = any(%s)
AND osm_id is null
""", (self.country, to_delete))
cur.execute("ANALYSE location_postcodes")
if to_update:
cur.executemany(
pysql.SQL("""UPDATE location_postcode
SET indexed_status = 2,
geometry = ST_SetSRID(ST_Point(%s, %s), 4326)
WHERE country_code = {} and postcode = %s
""").format(pysql.Literal(self.country)),
to_update)
def _compute_changes(
self, conn: Connection
) -> Tuple[List[Tuple[str, float, float]], List[str], List[Tuple[float, float, str]]]:
""" Compute which postcodes from the collected postcodes have to be
added or modified and which from the location_postcode table
have to be deleted.
"""
to_update = []
to_delete = []
with conn.cursor() as cur:
cur.execute("""SELECT postcode, ST_X(geometry), ST_Y(geometry)
FROM location_postcode
WHERE country_code = %s""",
(self.country, ))
for postcode, x, y in cur:
pcobj = self.collected.pop(postcode, None)
if pcobj:
newx, newy = pcobj.centroid()
if abs(x - newx) > 0.0000001 or abs(y - newy) > 0.0000001:
to_update.append((newx, newy, postcode))
else:
to_delete.append(postcode)
to_add = [(k, *v.centroid()) for k, v in self.collected.items()]
self.collected = defaultdict(PointsCentroid)
return to_add, to_delete, to_update
def _update_from_external(self, analyzer: AbstractAnalyzer, project_dir: Path) -> None:
""" Look for an external postcode file for the active country in
@@ -142,7 +152,7 @@ class _PostcodeCollector:
_to_float(row['lat'], 90))
self.collected[postcode] += centroid
except ValueError:
LOG.warning("Bad coordinates %s, %s in '%s' country postcode file.",
LOG.warning("Bad coordinates %s, %s in %s country postcode file.",
row['lat'], row['lon'], self.country)
finally:
@@ -165,159 +175,63 @@ class _PostcodeCollector:
def update_postcodes(dsn: str, project_dir: Optional[Path], tokenizer: AbstractTokenizer) -> None:
""" Update the table of postcodes from the input tables
placex and place_postcode.
""" Update the table of artificial postcodes.
Computes artificial postcode centroids from the placex table,
potentially enhances it with external data and then updates the
postcodes in the table 'location_postcode'.
"""
matcher = PostcodeFormatter()
with tokenizer.name_analyzer() as analyzer:
with connect(dsn) as conn:
# Backfill country_code column where required
conn.execute("""UPDATE place_postcode
SET country_code = get_country_code(centroid)
WHERE country_code is null
""")
# Now update first postcode areas
_update_postcode_areas(conn, analyzer, matcher)
# Then fill with estimated postcode centroids from other info
_update_guessed_postcode(conn, analyzer, matcher, project_dir)
# First get the list of countries that currently have postcodes.
# (Doing this before starting to insert, so it is fast on import.)
with conn.cursor() as cur:
cur.execute("SELECT DISTINCT country_code FROM location_postcode")
todo_countries = set((row[0] for row in cur))
# Recompute the list of valid postcodes from placex.
with conn.cursor(name="placex_postcodes") as cur:
cur.execute("""
SELECT cc, pc, ST_X(centroid), ST_Y(centroid)
FROM (SELECT
COALESCE(plx.country_code,
get_country_code(ST_Centroid(pl.geometry))) as cc,
pl.address->'postcode' as pc,
COALESCE(plx.centroid, ST_Centroid(pl.geometry)) as centroid
FROM place AS pl LEFT OUTER JOIN placex AS plx
ON pl.osm_id = plx.osm_id AND pl.osm_type = plx.osm_type
WHERE pl.address ? 'postcode' AND pl.geometry IS NOT null) xx
WHERE pc IS NOT null AND cc IS NOT null
ORDER BY cc, pc""")
collector = None
for country, postcode, x, y in cur:
if collector is None or country != collector.country:
if collector is not None:
collector.commit(conn, analyzer, project_dir)
collector = _PostcodeCollector(country, matcher.get_matcher(country))
todo_countries.discard(country)
collector.add(postcode, x, y)
if collector is not None:
collector.commit(conn, analyzer, project_dir)
# Now handle any countries that are only in the postcode table.
for country in todo_countries:
fmt = matcher.get_matcher(country)
_PostcodeCollector(country, fmt).commit(conn, analyzer, project_dir)
conn.commit()
analyzer.update_postcodes_from_db()
def _insert_postcode_areas(conn: Connection, country_code: str,
extent: int, pcs: list[dict[str, str]]) -> None:
if pcs:
with conn.cursor() as cur:
cur.executemany(
pysql.SQL(
""" INSERT INTO location_postcodes
(osm_id, country_code, rank_search, postcode, centroid, geometry)
SELECT osm_id, country_code, {}, %(out)s, centroid, geometry
FROM place_postcode
WHERE osm_type = 'R'
and country_code = {} and postcode = %(in)s
and geometry is not null
""").format(pysql.Literal(_extent_to_rank(extent)),
pysql.Literal(country_code)),
pcs)
def _update_postcode_areas(conn: Connection, analyzer: AbstractAnalyzer,
matcher: PostcodeFormatter) -> None:
""" Update the postcode areas made from postcode boundaries.
"""
# first delete all areas that have gone
conn.execute(""" DELETE FROM location_postcodes pc
WHERE pc.osm_id is not null
AND NOT EXISTS(
SELECT * FROM place_postcode pp
WHERE pp.osm_type = 'R' and pp.osm_id = pc.osm_id
and geometry is not null)
""")
# now insert all in country batches, triggers will ensure proper updates
with conn.cursor() as cur:
cur.execute(""" SELECT country_code, postcode FROM place_postcode
WHERE geometry is not null and osm_type = 'R'
ORDER BY country_code
""")
country_code = None
fmt = None
pcs = []
for cc, postcode in cur:
if country_code is None:
country_code = cc
fmt = matcher.get_matcher(country_code)
elif country_code != cc:
_insert_postcode_areas(conn, country_code,
matcher.get_postcode_extent(country_code), pcs)
country_code = cc
fmt = matcher.get_matcher(country_code)
pcs = []
if fmt is not None:
if (m := fmt.match(postcode)):
pcs.append({'out': fmt.normalize(m), 'in': postcode})
if country_code is not None and pcs:
_insert_postcode_areas(conn, country_code,
matcher.get_postcode_extent(country_code), pcs)
def _update_guessed_postcode(conn: Connection, analyzer: AbstractAnalyzer,
matcher: PostcodeFormatter, project_dir: Optional[Path]) -> None:
""" Computes artificial postcode centroids from the placex table,
potentially enhances it with external data and then updates the
postcodes in the table 'location_postcodes'.
"""
# First get the list of countries that currently have postcodes.
# (Doing this before starting to insert, so it is fast on import.)
with conn.cursor() as cur:
cur.execute("""SELECT DISTINCT country_code FROM location_postcodes
WHERE osm_id is null""")
todo_countries = {row[0] for row in cur}
# Next, get the list of postcodes that are already covered by areas.
area_pcs = defaultdict(set)
with conn.cursor() as cur:
cur.execute("""SELECT country_code, postcode
FROM location_postcodes WHERE osm_id is not null
ORDER BY country_code""")
for cc, pc in cur:
area_pcs[cc].add(pc)
# Create a temporary table which contains coverage of the postcode areas.
with conn.cursor() as cur:
cur.execute("DROP TABLE IF EXISTS _global_postcode_area")
cur.execute("""CREATE TABLE _global_postcode_area AS
(SELECT ST_SubDivide(ST_SimplifyPreserveTopology(
ST_Union(geometry), 0.00001), 128) as geometry
FROM place_postcode WHERE geometry is not null)
""")
cur.execute("CREATE INDEX ON _global_postcode_area USING gist(geometry)")
# Recompute the list of valid postcodes from placex.
with conn.cursor(name="placex_postcodes") as cur:
cur.execute("""
SELECT country_code, postcode, ST_X(centroid), ST_Y(centroid)
FROM (
(SELECT country_code, address->'postcode' as postcode, centroid
FROM placex WHERE address ? 'postcode')
UNION
(SELECT country_code, postcode, centroid
FROM place_postcode WHERE geometry is null)
) x
WHERE not postcode like '%,%' and not postcode like '%;%'
AND NOT EXISTS(SELECT * FROM _global_postcode_area g
WHERE ST_Intersects(x.centroid, g.geometry))
ORDER BY country_code""")
collector = None
for country, postcode, x, y in cur:
if collector is None or country != collector.country:
if collector is not None:
collector.commit(conn, analyzer, project_dir)
collector = _PostcodeCollector(country, matcher.get_matcher(country),
matcher.get_postcode_extent(country),
exclude=area_pcs[country])
todo_countries.discard(country)
collector.add(postcode, x, y)
if collector is not None:
collector.commit(conn, analyzer, project_dir)
# Now handle any countries that are only in the postcode table.
for country in todo_countries:
fmt = matcher.get_matcher(country)
ext = matcher.get_postcode_extent(country)
_PostcodeCollector(country, fmt, ext,
exclude=area_pcs[country]).commit(conn, analyzer, project_dir)
conn.execute("DROP TABLE IF EXISTS _global_postcode_area")
def can_compute(dsn: str) -> bool:
""" Check that the necessary tables exist so that postcodes can be computed.
"""
Check that the place table exists so that
postcodes can be computed.
"""
with connect(dsn) as conn:
return table_exists(conn, 'place_postcode')
return table_exists(conn, 'place')

View File

@@ -11,8 +11,6 @@ from typing import Iterable
import re
import logging
import mwparserfromhell
from ...config import Configuration
from ...utils.url_utils import get_url
from .special_phrase import SpecialPhrase
@@ -38,6 +36,10 @@ class SPWikiLoader:
"""
def __init__(self, config: Configuration) -> None:
self.config = config
# Compile the regex here to increase performances.
self.occurence_pattern = re.compile(
r'\| *([^\|]+) *\|\| *([^\|]+) *\|\| *([^\|]+) *\|\| *([^\|]+) *\|\| *([\-YN])'
)
# Hack around a bug where building=yes was imported with quotes into the wiki
self.type_fix_pattern = re.compile(r'\"|&quot;')
@@ -56,21 +58,11 @@ class SPWikiLoader:
LOG.warning('Importing phrases for lang: %s...', lang)
loaded_xml = _get_wiki_content(lang)
wikicode = mwparserfromhell.parse(loaded_xml)
# One match will be of format [label, class, type, operator, plural]
matches = self.occurence_pattern.findall(loaded_xml)
for table in wikicode.filter_tags(matches=lambda t: t.tag == 'table'):
for row in table.contents.filter_tags(matches=lambda t: t.tag == 'tr'):
cells = list(row.contents.filter_tags(matches=lambda t: t.tag == 'td'))
if len(cells) < 5:
continue
label = cells[0].contents.strip_code().strip()
cls = cells[1].contents.strip_code().strip()
typ = cells[2].contents.strip_code().strip()
operator = cells[3].contents.strip_code().strip()
yield SpecialPhrase(label,
cls,
self.type_fix_pattern.sub('', typ),
operator)
for match in matches:
yield SpecialPhrase(match[0],
match[1],
self.type_fix_pattern.sub('', match[2]),
match[3])

View File

@@ -17,12 +17,13 @@ import tarfile
from psycopg.types.json import Json
from ..config import Configuration
from ..db.connection import connect, table_exists
from ..db.connection import connect
from ..db.sql_preprocessor import SQLPreprocessor
from ..errors import UsageError
from ..db.query_pool import QueryPool
from ..data.place_info import PlaceInfo
from ..tokenizer.base import AbstractTokenizer
from . import freeze
LOG = logging.getLogger()
@@ -89,19 +90,16 @@ async def add_tiger_data(data_dir: str, config: Configuration, threads: int,
"""
dsn = config.get_libpq_dsn()
with connect(dsn) as conn:
if freeze.is_frozen(conn):
raise UsageError("Tiger cannot be imported when database frozen (Github issue #3048)")
with TigerInput(data_dir) as tar:
if not tar:
return 1
with connect(dsn) as conn:
sql = SQLPreprocessor(conn, config)
if not table_exists(conn, 'search_name'):
raise UsageError(
"Cannot perform tiger import: required tables are missing. "
"See https://github.com/osm-search/Nominatim/issues/2463 for details."
)
sql.run_sql_file(conn, 'tiger_import_start.sql')
# Reading files and then for each file line handling

View File

@@ -55,7 +55,7 @@ def parse_version(version: str) -> NominatimVersion:
return NominatimVersion(*[int(x) for x in parts[:2] + parts[2].split('-')])
NOMINATIM_VERSION = parse_version('5.2.99-2')
NOMINATIM_VERSION = parse_version('5.2.0-0')
POSTGRESQL_REQUIRED_VERSION = (12, 0)
POSTGIS_REQUIRED_VERSION = (3, 0)

View File

@@ -9,7 +9,6 @@ Fixtures for BDD test steps
"""
import sys
import json
import re
from pathlib import Path
import psycopg
@@ -21,8 +20,7 @@ sys.path.insert(0, str(SRC_DIR / 'src'))
import pytest
from pytest_bdd.parsers import re as step_parse
from pytest_bdd import given, when, then, scenario
from pytest_bdd.feature import get_features
from pytest_bdd import given, when, then
pytest.register_assert_rewrite('utils')
@@ -375,57 +373,3 @@ def check_place_missing_lines(db_conn, table, osm_type, osm_id, osm_class):
with db_conn.cursor() as cur:
assert cur.execute(sql, params).fetchone()[0] == 0
if pytest.version_tuple >= (8, 0, 0):
def pytest_pycollect_makemodule(module_path, parent):
return BddTestCollector.from_parent(parent, path=module_path)
class BddTestCollector(pytest.Module):
def __init__(self, **kwargs):
super().__init__(**kwargs)
def collect(self):
for item in super().collect():
yield item
if hasattr(self.obj, 'PYTEST_BDD_SCENARIOS'):
for path in self.obj.PYTEST_BDD_SCENARIOS:
for feature in get_features([str(Path(self.path.parent, path).resolve())]):
yield FeatureFile.from_parent(self,
name=str(Path(path, feature.rel_filename)),
path=Path(feature.filename),
feature=feature)
# borrowed from pytest-bdd: src/pytest_bdd/scenario.py
def make_python_name(string: str) -> str:
"""Make python attribute name out of a given string."""
string = re.sub(r"\W", "", string.replace(" ", "_"))
return re.sub(r"^\d+_*", "", string).lower()
class FeatureFile(pytest.File):
class obj:
pass
def __init__(self, feature, **kwargs):
self.feature = feature
super().__init__(**kwargs)
def collect(self):
for sname, sobject in self.feature.scenarios.items():
class_name = f"L{sobject.line_number}"
test_name = "test_" + make_python_name(sname)
@scenario(self.feature.filename, sname)
def _test():
pass
tclass = type(class_name, (),
{test_name: staticmethod(_test)})
setattr(self.obj, class_name, tclass)
yield pytest.Class.from_parent(self, name=class_name, obj=tclass)

View File

@@ -42,22 +42,6 @@ Feature: Tests for finding places by osm_type and osm_id
| jsonv2 | json |
| geojson | geojson |
Scenario Outline: Lookup with entrances
When sending v1/lookup with format <format>
| osm_ids | entrances |
| W429210603 | 1 |
Then a HTTP 200 is returned
And the result is valid <outformat>
And result 0 contains in field entrances+0
| osm_id | type | lat | lon |
| 6580031131 | yes | 47.2489382 | 9.5284033 |
Examples:
| format | outformat |
| json | json |
| jsonv2 | json |
| geojson | geojson |
Scenario: Linked places return information from the linkee
When sending v1/lookup with format geocodejson
| osm_ids |

View File

@@ -167,18 +167,3 @@ Feature: v1/reverse Parameter Tests
| json | json |
| jsonv2 | json |
| xml | xml |
Scenario Outline: Reverse with entrances
When sending v1/reverse with format <format>
| lat | lon | entrances | zoom |
| 47.24942041089678 | 9.52854573737568 | 1 | 18 |
Then a HTTP 200 is returned
And the result is valid <outformat>
And the result contains array field entrances where element 0 contains
| osm_id | type | lat | lon |
| 6580031131 | yes | 47.2489382 | 9.5284033 |
Examples:
| format | outformat |
| json | json |
| jsonv2 | json |

View File

@@ -268,6 +268,33 @@ Feature: Address computation
| W93 | R34 |
| W93 | R4 |
Scenario: postcode boundaries do appear in the address of a way
Given the grid with origin DE
| 1 | | | | | 8 | | 6 | | 2 |
| |10 |11 | | | | | | | |
| |13 |12 | | | | | | | |
| 20| | | 21| | | | | | |
| | | | | | | | | | |
| | | | | | 9 | | | | |
| 4 | | | | | | | 7 | | 3 |
And the named places
| osm | class | type | admin | addr+postcode | geometry |
| R1 | boundary | administrative | 6 | 10000 | (1,2,3,4,1) |
| R34 | boundary | administrative | 8 | 11000 | (1,6,7,4,1) |
And the places
| osm | class | type | addr+postcode | geometry |
| R4 | boundary | postal_code | 11200 | (1,8,9,4,1) |
And the named places
| osm | class | type | geometry |
| W93 | highway | residential | 20,21 |
And the places
| osm | class | type | addr+postcode | geometry |
| W22 | place | postcode | 11234 | (10,11,12,13,10) |
When importing
Then place_addressline contains
| object | address |
| W93 | R4 |
Scenario: squares do not appear in the address of a street
Given the grid
| | 1 | | 2 | |

View File

@@ -16,6 +16,21 @@ Feature: Linking of places
| R13 | - |
| N256 | - |
Scenario: Postcode areas cannot be linked
Given the grid with origin US
| 1 | | 2 |
| | 9 | |
| 4 | | 3 |
And the named places
| osm | class | type | addr+postcode | extra+wikidata | geometry |
| R13 | boundary | postal_code | 12345 | Q87493 | (1,2,3,4,1) |
| N25 | place | suburb | 12345 | Q87493 | 9 |
When importing
Then placex contains
| object | linked_place_id |
| R13 | - |
| N25 | - |
Scenario: Waterways are linked when in waterway relations
Given the grid
| 1 | | | | 3 | 4 | | | | 6 |
@@ -306,17 +321,11 @@ Feature: Linking of places
Given the places
| osm | class | type | name+name | geometry |
| N9 | place | city | Popayán | 9 |
Given the places
| osm | class | type | name+name | geometry | admin |
| R1 | boundary | administrative | Perímetro Urbano Popayán | (1,2,3,4,1) | 8 |
| R1 | boundary | administrative | Perímetro Urbano Popayán | (1,2,3,4,1) |
And the relations
| id | members |
| 1 | N9:label |
When importing
Then placex contains
| object | linked_place_id |
| N9:place | R1 |
| R1:boundary | - |
Then placex contains
| object | name+_place_name | name+_place_name:es |
| R1 | Popayán | Popayán |

View File

@@ -46,6 +46,23 @@ Feature: Import into placex
| object | admin_level |
| N1 | 3 |
Scenario: postcode node without postcode is dropped
Given the places
| osm | class | type | name+ref |
| N1 | place | postcode | 12334 |
When importing
Then placex has no entry for N1
Scenario: postcode boundary without postcode is dropped
Given the 0.01 grid
| 1 | 2 |
| 3 | |
Given the places
| osm | class | type | name+ref | geometry |
| R1 | boundary | postal_code | 554476 | (1,2,3,1) |
When importing
Then placex has no entry for R1
Scenario: search and address ranks for boundaries are correctly assigned
Given the named places
| osm | class | type |

View File

@@ -121,8 +121,8 @@ Feature: Import of postcodes
| | 1 | 2 | | |
| | 4 | 3 | | |
And the named places
| osm | class | type | geometry |
| W93 | highway | pedestrian | (10,11,12,13,10) |
| osm | class | type | geometry |
| W93 | highway | pedestriant | (10,11,12,13,10) |
And the named places
| osm | class | type | addr+postcode | geometry |
| W22 | building | yes | 45023 | (1,2,3,4,1) |
@@ -134,13 +134,14 @@ Feature: Import of postcodes
Scenario: Roads get postcodes from nearby unnamed buildings without other info
Given the grid with origin US
| 10 | | | | 11 |
| | 1 | | | |
| | 1 | 2 | | |
| | 4 | 3 | | |
And the named places
| osm | class | type | geometry |
| W93 | highway | residential | 10,11 |
And the postcodes
| osm | postcode | centroid |
| W22 | 45023 | 1 |
And the places
| osm | class | type | addr+postcode | geometry |
| W22 | place | postcode | 45023 | (1,2,3,4,1) |
When importing
Then placex contains
| object | postcode |
@@ -171,12 +172,26 @@ Feature: Import of postcodes
Scenario: Postcodes are added to the postcode
Given the places
| osm | class | type | addr+postcode | addr+housenumber | geometry |
| N34 | place | house | 01982 | 111 | country:de |
| N34 | place | house | 01982 | 111 |country:de |
When importing
Then location_postcodes contains exactly
| country_code | postcode | centroid!wkt |
Then location_postcode contains exactly
| country_code | postcode | geometry!wkt |
| de | 01982 | country:de |
@skip
Scenario: search and address ranks for GB post codes correctly assigned
Given the places
| osm | class | type | postcode | geometry |
| N1 | place | postcode | E45 2CD | country:gb |
| N2 | place | postcode | E45 2 | country:gb |
| N3 | place | postcode | Y45 | country:gb |
When importing
Then location_postcode contains exactly
| postcode | country_code | rank_search | rank_address |
| E45 2CD | gb | 25 | 5 |
| E45 2 | gb | 23 | 5 |
| Y45 | gb | 21 | 5 |
Scenario: Postcodes outside all countries are not added to the postcode table
Given the places
| osm | class | type | addr+postcode | addr+housenumber | addr+place | geometry |
@@ -185,7 +200,7 @@ Feature: Import of postcodes
| osm | class | type | name | geometry |
| N1 | place | hamlet | Null Island | 0 0 |
When importing
Then location_postcodes contains exactly
Then location_postcode contains exactly
| place_id |
When geocoding "111, 01982 Null Island"
Then the result set contains

View File

@@ -154,6 +154,19 @@ Feature: Import and search of names
| object |
| R2 |
Scenario: Postcode boundaries without ref
Given the grid with origin FR
| | 2 | |
| 1 | | 3 |
Given the places
| osm | class | type | postcode | geometry |
| R1 | boundary | postal_code | 123-45 | (1,2,3,1) |
When importing
When geocoding "123-45"
Then result 0 contains
| object |
| R1 |
Scenario Outline: Housenumbers with special characters are found
Given the grid
| 1 | | | | 2 |

View File

@@ -78,8 +78,8 @@ Feature: Querying fo postcode variants
| N34 | place | house | EH4 7EA | 111 | country:gb |
| N35 | place | house | E4 7EA | 111 | country:gb |
When importing
Then location_postcodes contains exactly
| country_code | postcode | centroid!wkt |
Then location_postcode contains exactly
| country_code | postcode | geometry!wkt |
| gb | EH4 7EA | country:gb |
| gb | E4 7EA | country:gb |
When geocoding "EH4 7EA"
@@ -90,3 +90,20 @@ Feature: Querying fo postcode variants
Then result 0 contains
| type | display_name |
| postcode | E4 7EA, United Kingdom |
Scenario: Postcode areas are preferred over postcode points
Given the grid with origin DE
| 1 | 2 |
| 4 | 3 |
Given the places
| osm | class | type | postcode | geometry |
| R23 | boundary | postal_code | 12345 | (1,2,3,4,1) |
When importing
Then location_postcode contains exactly
| country_code | postcode |
| de | 12345 |
When geocoding "12345, de"
Then result 0 contains
| object |
| R23 |

View File

@@ -9,32 +9,13 @@ Feature: Reverse searches
And the places
| osm | class | type | geometry |
| W1 | aeroway | terminal | (1,2,3,4,1) |
| N9 | amenity | restaurant | 9 |
| N1 | amenity | restaurant | 9 |
When importing
And reverse geocoding 1.0001,1.0001
Then the result contains
| object |
| N9 |
| N1 |
When reverse geocoding 1.0003,1.0001
Then the result contains
| object |
| W1 |
Scenario: Find closest housenumber for street matches
Given the 0.0001 grid with origin 1,1
| | 1 | | |
| | | 2 | |
| 10 | | | 11 |
And the places
| osm | class | type | name | geometry |
| W1 | highway | service | Goose Drive | 10,11 |
| N2 | tourism | art_work | Beauty | 2 |
And the places
| osm | class | type | housenr | geometry |
| N1 | place | house | 23 | 1 |
When importing
When reverse geocoding 1.0002,1.0002
Then the result contains
| object |
| N1 |

View File

@@ -0,0 +1,20 @@
Feature: Update of names in place objects
Test all naming related issues in updates
Scenario: Delete postcode from postcode boundaries without ref
Given the grid with origin DE
| 1 | 2 |
| 4 | 3 |
Given the places
| osm | class | type | postcode | geometry |
| R1 | boundary | postal_code | 123-45 | (1,2,3,4,1) |
When importing
And geocoding "123-45"
Then result 0 contains
| object |
| R1 |
When updating places
| osm | class | type | geometry |
| R1 | boundary | postal_code | (1,2,3,4,1) |
Then placex has no entry for R1

View File

@@ -2,22 +2,20 @@ Feature: Update of postcode
Tests for updating of data related to postcodes
Scenario: Updating postcode in postcode boundaries without ref
Given the grid with origin FR
| 1 | | 2 |
| | 9 | |
| 4 | | 3 |
Given the postcodes
| osm | postcode | centroid | geometry |
| R1 | 12345 | 9 | (1,2,3,4,1) |
Given the grid
| 1 | 2 |
| 4 | 3 |
Given the places
| osm | class | type | postcode | geometry |
| R1 | boundary | postal_code | 12345 | (1,2,3,4,1) |
When importing
And geocoding "12345"
Then result 0 contains
| object |
| R1 |
Given the postcodes
| osm | postcode | centroid | geometry |
| R1 | 54321 | 9 | (1,2,3,4,1) |
When refreshing postcodes
When updating places
| osm | class | type | postcode | geometry |
| R1 | boundary | postal_code | 54321 | (1,2,3,4,1) |
And geocoding "12345"
Then exactly 0 results are returned
When geocoding "54321"
@@ -30,21 +28,17 @@ Feature: Update of postcode
| osm | class | type | addr+postcode | addr+housenumber | geometry |
| N34 | place | house | 01982 | 111 | country:de |
When importing
Then location_postcodes contains exactly
| country_code | postcode | centroid!wkt |
Then location_postcode contains exactly
| country_code | postcode | geometry!wkt |
| de | 01982 | country:de |
Given the postcodes
| osm | postcode | centroid |
| N66 | 99201 | country:fr |
When updating places
| osm | class | type | addr+postcode | addr+housenumber | geometry |
| N35 | place | house | 4567 | 5 | country:ch |
And refreshing postcodes
Then location_postcodes contains exactly
| country_code | postcode | centroid!wkt |
And updating postcodes
Then location_postcode contains exactly
| country_code | postcode | geometry!wkt |
| de | 01982 | country:de |
| ch | 4567 | country:ch |
| fr | 99201 | country:fr |
Scenario: When the last postcode is deleted, it is deleted from postcode
Given the places
@@ -53,9 +47,9 @@ Feature: Update of postcode
| N35 | place | house | 4567 | 5 | country:ch |
When importing
And marking for delete N34
And refreshing postcodes
Then location_postcodes contains exactly
| country_code | postcode | centroid!wkt |
And updating postcodes
Then location_postcode contains exactly
| country_code | postcode | geometry!wkt |
| ch | 4567 | country:ch |
Scenario: A postcode is not deleted from postcode when it exist in another country
@@ -65,24 +59,64 @@ Feature: Update of postcode
| N35 | place | house | 01982 | 5 | country:fr |
When importing
And marking for delete N34
And refreshing postcodes
Then location_postcodes contains exactly
| country_code | postcode | centroid!wkt|
And updating postcodes
Then location_postcode contains exactly
| country_code | postcode | geometry!wkt|
| fr | 01982 | country:fr |
Scenario: Updating a postcode is reflected in postcode table
Given the places
| osm | class | type | addr+postcode | geometry |
| osm | class | type | addr+postcode | geometry |
| N34 | place | postcode | 01982 | country:de |
When importing
And updating places
| osm | class | type | addr+postcode | geometry |
| N34 | place | postcode | 20453 | country:de |
And refreshing postcodes
Then location_postcodes contains exactly
| country_code | postcode | centroid!wkt |
And updating postcodes
Then location_postcode contains exactly
| country_code | postcode | geometry!wkt |
| de | 20453 | country:de |
Scenario: When changing from a postcode type, the entry appears in placex
When importing
And updating places
| osm | class | type | addr+postcode | geometry |
| N34 | place | postcode | 01982 | country:de |
Then placex has no entry for N34
When updating places
| osm | class | type | addr+postcode | housenr | geometry |
| N34 | place | house | 20453 | 1 | country:de |
Then placex contains
| object | addr+housenumber | geometry!wkt |
| N34 | 1 | country:de |
And place contains exactly
| osm_type | osm_id | class | type |
| N | 34 | place | house |
When updating postcodes
Then location_postcode contains exactly
| country_code | postcode | geometry!wkt |
| de | 20453 | country:de |
Scenario: When changing to a postcode type, the entry disappears from placex
When importing
And updating places
| osm | class | type | addr+postcode | housenr | geometry |
| N34 | place | house | 20453 | 1 | country:de |
Then placex contains
| object | addr+housenumber | geometry!wkt |
| N34 | 1 | country:de|
When updating places
| osm | class | type | addr+postcode | geometry |
| N34 | place | postcode | 01982 | country:de |
Then placex has no entry for N34
And place contains exactly
| osm_type | osm_id | class | type |
| N | 34 | place | postcode |
When updating postcodes
Then location_postcode contains exactly
| country_code | postcode | geometry!wkt |
| de | 01982 | country:de |
Scenario: When a parent is deleted, the postcode gets a new parent
Given the grid with origin DE
| 1 | | 3 | 4 |
@@ -92,59 +126,14 @@ Feature: Update of postcode
| osm | class | type | name | admin | geometry |
| R1 | boundary | administrative | Big | 6 | (1,4,6,2,1) |
| R2 | boundary | administrative | Small | 6 | (1,3,5,2,1) |
Given the postcodes
| osm | postcode | centroid |
| N9 | 12345 | 9 |
Given the places
| osm | class | type | addr+postcode | geometry |
| N9 | place | postcode | 12345 | 9 |
When importing
Then location_postcodes contains exactly
| postcode | centroid!wkt | parent_place_id |
Then location_postcode contains exactly
| postcode | geometry!wkt | parent_place_id |
| 12345 | 9 | R2 |
When marking for delete R2
Then location_postcodes contains exactly
| country_code | postcode | centroid!wkt | parent_place_id |
Then location_postcode contains exactly
| country_code | postcode | geometry!wkt | parent_place_id |
| de | 12345 | 9 | R1 |
Scenario: When a postcode area appears, postcode points are shadowed
Given the grid with origin DE
| 1 | | 3 | |
| | 9 | | 8 |
| 2 | | 5 | |
Given the postcodes
| osm | postcode | centroid |
| N92 | 44321 | 9 |
| N4 | 00245 | 8 |
When importing
Then location_postcodes contains exactly
| country_code | postcode | osm_id | centroid!wkt |
| de | 44321 | - | 9 |
| de | 00245 | - | 8 |
Given the postcodes
| osm | postcode | centroid | geometry |
| R45 | 00245 | 9 | (1,3,5,2,1) |
When refreshing postcodes
Then location_postcodes contains exactly
| country_code | postcode | osm_id | centroid!wkt |
| de | 00245 | 45 | 9 |
Scenario: When a postcode area disappears, postcode points are unshadowed
Given the grid with origin DE
| 1 | | 3 | |
| | 9 | | 8 |
| 2 | | 5 | |
Given the postcodes
| osm | postcode | centroid | geometry |
| R45 | 00245 | 9 | (1,3,5,2,1) |
Given the postcodes
| osm | postcode | centroid |
| N92 | 44321 | 9 |
| N4 | 00245 | 8 |
When importing
Then location_postcodes contains exactly
| country_code | postcode | osm_id | centroid!wkt |
| de | 00245 | 45 | 9 |
When marking for delete R45
And refreshing postcodes
Then location_postcodes contains exactly
| country_code | postcode | osm_id | centroid!wkt |
| de | 44321 | - | 9 |
| de | 00245 | - | 8 |

View File

@@ -92,16 +92,12 @@ Feature: Tag evaluation
n6001 Tshop=bank,addr:postcode=12345
n6002 Tshop=bank,tiger:zip_left=34343
n6003 Tshop=bank,is_in:postcode=9009
n6004 Taddr:postcode=54322
"""
Then place contains exactly
| object | class | address!dict |
| N6001 | shop | 'postcode': '12345' |
| N6002 | shop | 'postcode': '34343' |
| N6003 | shop | - |
And place_postcode contains exactly
| object | postcode | geometry |
| N6004 | 54322 | - |
Scenario: Postcode areas
@@ -111,15 +107,11 @@ Feature: Tag evaluation
n2 x12.36853 y51.42362
n3 x12.63666 y51.42362
n4 x12.63666 y51.50618
w1 Nn1,n2,n3,n4,n1
w2 Tboundary=postal_code,postal_code=443 Nn1,n2,n3,n4,n1
r1 Ttype=boundary,boundary=postal_code,postal_code=3456 Mw1@
w1 Tboundary=postal_code,ref=3456 Nn1,n2,n3,n4,n1
"""
Then place contains exactly
| object |
And place_postcode contains exactly
| object | postcode | geometry!wkt |
| R1 | 3456 | (12.36853 51.50618, 12.36853 51.42362, 12.63666 51.42362, 12.63666 51.50618, 12.36853 51.50618) |
| object | class | type | name!dict |
| W1 | boundary | postal_code | 'ref': '3456' |
Scenario: Main with extra
When loading osm data
@@ -200,9 +192,7 @@ Feature: Tag evaluation
| N12001 | tourism | hotel |
| N12003 | building | shed |
| N12004 | building | yes |
And place_postcode contains exactly
| object | postcode | geometry |
| N12005 | 12345 | - |
| N12005 | place | postcode |
Scenario: Address interpolations

View File

@@ -2,6 +2,7 @@ Feature: Update of postcode only objects
Tests that changes to objects containing only a postcode are
propagated correctly.
Scenario: Adding a postcode-only node
When loading osm data
"""
@@ -14,10 +15,11 @@ Feature: Update of postcode only objects
"""
n34 Tpostcode=4456
"""
Then place_postcode contains exactly
| object | postcode |
| N34 | 4456 |
And place contains exactly
Then place contains exactly
| object | class | type |
| N34 | place | postcode |
When indexing
Then placex contains exactly
| object |
@@ -26,11 +28,9 @@ Feature: Update of postcode only objects
"""
n34 Tpostcode=4456
"""
Then place_postcode contains exactly
| object | postcode |
| N34 | 4456 |
And place contains exactly
| object |
Then place contains exactly
| object | class | type |
| N34 | place | postcode |
When updating osm data
"""
@@ -38,7 +38,8 @@ Feature: Update of postcode only objects
"""
Then place contains exactly
| object |
And place_postcode contains exactly
When indexing
Then placex contains exactly
| object |
@@ -56,10 +57,8 @@ Feature: Update of postcode only objects
n34 Tpostcode=4456
"""
Then place contains exactly
| object |
And place_postcode contains exactly
| object | postcode |
| N34 | 4456 |
| object | class | type |
| N34 | place | postcode |
When indexing
Then placex contains exactly
| object |
@@ -75,9 +74,9 @@ Feature: Update of postcode only objects
"""
n34 Tpostcode=4456
"""
Then place_postcode contains exactly
| object | postcode |
| N34 | 4456 |
Then place contains exactly
| object | class | type |
| N34 | place | postcode |
When updating osm data
"""
@@ -86,8 +85,6 @@ Feature: Update of postcode only objects
Then place contains exactly
| object | class | type |
| N34 | <class> | <type> |
And place_postcode contains exactly
| object |
When indexing
Then placex contains exactly
| object | class | type |
@@ -99,7 +96,7 @@ Feature: Update of postcode only objects
| place | hamlet |
Scenario: Converting an interpolation into a postcode-only node
Scenario: Converting na interpolation into a postcode-only node
Given the grid
| 1 | 2 |
When loading osm data
@@ -122,12 +119,14 @@ Feature: Update of postcode only objects
| object | class | type |
| N1 | place | house |
| N2 | place | house |
Then place_postcode contains exactly
| object | postcode |
| W34 | 4456 |
| W34 | place | postcode |
When indexing
Then location_property_osmline contains exactly
| osm_id |
And placex contains exactly
| object | class | type |
| N1 | place | house |
| N2 | place | house |
Scenario: Converting a postcode-only node into an interpolation
@@ -145,9 +144,7 @@ Feature: Update of postcode only objects
| N1 | place | house |
| N2 | place | house |
| W33 | highway | residential |
And place_postcode contains exactly
| object | postcode |
| W34 | 4456 |
| W34 | place | postcode |
When updating osm data
"""
@@ -159,8 +156,6 @@ Feature: Update of postcode only objects
| N2 | place | house |
| W33 | highway | residential |
| W34 | place | houses |
And place_postcode contains exactly
| object |
When indexing
Then location_property_osmline contains exactly
| osm_id | startnumber | endnumber |

View File

@@ -15,7 +15,7 @@ import xml.etree.ElementTree as ET
import pytest
from pytest_bdd.parsers import re as step_parse
from pytest_bdd import when, given, then
from pytest_bdd import scenarios, when, given, then
from nominatim_db import cli
from nominatim_db.config import Configuration
@@ -150,8 +150,4 @@ def parse_api_json_response(api_response, fmt, num):
return result
if pytest.version_tuple >= (8, 0, 0):
PYTEST_BDD_SCENARIOS = ['features/api']
else:
from pytest_bdd import scenarios
scenarios('features/api')
scenarios('features/api')

View File

@@ -11,17 +11,15 @@ These tests check the Nominatim import chain after the osm2pgsql import.
"""
import asyncio
import re
from collections import defaultdict
import psycopg
import pytest
from pytest_bdd import when, then, given
from pytest_bdd import scenarios, when, then, given
from pytest_bdd.parsers import re as step_parse
from utils.place_inserter import PlaceColumn
from utils.checks import check_table_content
from utils.geometry_alias import ALIASES
from nominatim_db.config import Configuration
from nominatim_db import cli
@@ -99,41 +97,6 @@ def import_place_entrances(db_conn, datatable, node_grid):
data.columns.get('extratags')))
@given(step_parse('the postcodes'), target_fixture=None)
def import_place_postcode(db_conn, datatable, node_grid):
""" Insert todo rows into the place_postcode table. If a row for the
requested object already exists it is overwritten.
"""
with db_conn.cursor() as cur:
for row in datatable[1:]:
data = defaultdict(lambda: None)
data.update((k, v) for k, v in zip(datatable[0], row))
if data['centroid'].startswith('country:'):
ccode = data['centroid'][8:].upper()
data['centroid'] = 'srid=4326;POINT({} {})'.format(*ALIASES[ccode])
else:
data['centroid'] = f"srid=4326;{node_grid.geometry_to_wkt(data['centroid'])}"
data['osm_type'] = data['osm'][0]
data['osm_id'] = data['osm'][1:]
if 'geometry' in data:
geom = f"'srid=4326;{node_grid.geometry_to_wkt(data['geometry'])}'::geometry"
else:
geom = 'null'
cur.execute(""" DELETE FROM place_postcode
WHERE osm_type = %(osm_type)s and osm_id = %(osm_id)s""",
data)
cur.execute(f"""INSERT INTO place_postcode
(osm_type, osm_id, country_code, postcode, centroid, geometry)
VALUES (%(osm_type)s, %(osm_id)s,
%(country)s, %(postcode)s,
%(centroid)s, {geom})""", data)
db_conn.commit()
@given('the ways', target_fixture=None)
def import_ways(db_conn, datatable):
""" Import raw ways into the osm2pgsql way middle table.
@@ -205,7 +168,7 @@ def do_update(db_conn, update_config, node_grid, datatable):
@when('updating entrances', target_fixture=None)
def update_place_entrances(db_conn, datatable, node_grid):
""" Update rows in the place_entrance table.
""" Insert todo rows into the place_entrance table.
"""
with db_conn.cursor() as cur:
for row in datatable[1:]:
@@ -218,10 +181,9 @@ def update_place_entrances(db_conn, datatable, node_grid):
VALUES (%s, %s, %s, {})""".format(data.get_wkt()),
(data.columns['osm_id'], data.columns['type'],
data.columns.get('extratags')))
db_conn.commit()
@when('refreshing postcodes')
@when('updating postcodes')
def do_postcode_update(update_config):
""" Recompute the postcode centroids.
"""
@@ -241,8 +203,6 @@ def do_delete_place(db_conn, update_config, node_grid, otype, oid):
if otype == 'N':
cur.execute('DELETE FROM place_entrance WHERE osm_id = %s',
(oid, ))
cur.execute('DELETE FROM place_postcode WHERE osm_type = %s and osm_id = %s',
(otype, oid))
db_conn.commit()
cli.nominatim(['index', '-q'], update_config.environ)
@@ -316,8 +276,4 @@ def then_check_interpolation_table_negative(db_conn, oid):
assert cur.fetchone()[0] == 0
if pytest.version_tuple >= (8, 0, 0):
PYTEST_BDD_SCENARIOS = ['features/db']
else:
from pytest_bdd import scenarios
scenarios('features/db')
scenarios('features/db')

View File

@@ -11,7 +11,7 @@ import asyncio
import random
import pytest
from pytest_bdd import when, then, given
from pytest_bdd import scenarios, when, then, given
from pytest_bdd.parsers import re as step_parse
from nominatim_db import cli
@@ -106,8 +106,4 @@ def check_place_content(db_conn, datatable, node_grid, table, exact):
check_table_content(db_conn, table, datatable, grid=node_grid, exact=bool(exact))
if pytest.version_tuple >= (8, 0, 0):
PYTEST_BDD_SCENARIOS = ['features/osm2pgsql']
else:
from pytest_bdd import scenarios
scenarios('features/osm2pgsql')
scenarios('features/osm2pgsql')

View File

@@ -2,7 +2,7 @@
#
# This file is part of Nominatim. (https://nominatim.org)
#
# Copyright (C) 2026 by the Nominatim developer community.
# Copyright (C) 2025 by the Nominatim developer community.
# For a full list of authors see the git log.
"""
Various helper classes for running Nominatim commands.
@@ -54,14 +54,15 @@ class APIRunner:
def create_engine_starlette(self, environ):
import nominatim_api.server.starlette.server
from asgi_lifespan import LifespanManager
from starlette.testclient import TestClient
import httpx
async def _request(endpoint, params, http_headers):
app = nominatim_api.server.starlette.server.get_application(None, environ)
async with LifespanManager(app):
client = TestClient(app, base_url="http://nominatim.test")
response = client.get("/" + endpoint, params=params, headers=http_headers)
async with httpx.AsyncClient(app=app, base_url="http://nominatim.test") as client:
response = await client.get("/" + endpoint, params=params,
headers=http_headers)
return APIResponse(endpoint, response.status_code,
response.text, response.headers)

View File

@@ -7,7 +7,6 @@
"""
Helper functions to compare expected values.
"""
import ast
import collections.abc
import json
import re
@@ -59,8 +58,7 @@ COMPARISON_FUNCS = {
None: lambda val, exp: str(val) == exp,
'i': lambda val, exp: str(val).lower() == exp.lower(),
'fm': lambda val, exp: re.fullmatch(exp, val) is not None,
'dict': lambda val, exp: (val is None if exp == '-'
else (val == ast.literal_eval('{' + exp + '}'))),
'dict': lambda val, exp: val is None if exp == '-' else (val == eval('{' + exp + '}')),
'in_box': within_box
}

View File

@@ -8,7 +8,6 @@
A grid describing node placement in an area.
Useful for visually describing geometries.
"""
import re
class Grid:
@@ -45,28 +44,3 @@ class Grid:
def parse_line(self, value):
return [self.parse_point(p) for p in value.split(',')]
def geometry_to_wkt(self, value):
""" Parses the given value into a geometry and returns the WKT.
The value can either be a WKT already or a geometry shortcut
with coordinates or grid points.
"""
if re.fullmatch(r'([A-Z]+)\((.*)\)', value) is not None:
return value # already a WKT
# points
if ',' not in value:
x, y = self.parse_point(value)
return f"POINT({x} {y})"
# linestring
if '(' not in value:
coords = ','.join(' '.join(f"{p:.7f}" for p in pt)
for pt in self.parse_line(value))
return f"LINESTRING({coords})"
# simple polygons
coords = ','.join(' '.join(f"{p:.7f}" for p in pt)
for pt in self.parse_line(value[1:-1]))
return f"POLYGON(({coords}))"

View File

@@ -2,17 +2,15 @@
#
# This file is part of Nominatim. (https://nominatim.org)
#
# Copyright (C) 2026 by the Nominatim developer community.
# Copyright (C) 2025 by the Nominatim developer community.
# For a full list of authors see the git log.
"""
Helper classes for filling the place table.
"""
import ast
import random
import string
from .geometry_alias import ALIASES
from .grid import Grid
class PlaceColumn:
@@ -21,7 +19,7 @@ class PlaceColumn:
"""
def __init__(self, grid=None):
self.columns = {'admin_level': 15}
self.grid = grid or Grid()
self.grid = grid
self.geometry = None
def add_row(self, headings, row, force_name):
@@ -36,8 +34,7 @@ class PlaceColumn:
self._add_hstore(
'name',
'name',
''.join(random.choices(string.ascii_uppercase)
+ random.choices(string.printable, k=random.randrange(30))),
''.join(random.choices(string.printable, k=random.randrange(30))),
)
return self
@@ -52,7 +49,7 @@ class PlaceColumn:
elif key.startswith('addr+'):
self._add_hstore('address', key[5:], value)
elif key in ('name', 'address', 'extratags'):
self.columns[key] = ast.literal_eval('{' + value + '}')
self.columns[key] = eval('{' + value + '}')
else:
assert key in ('class', 'type'), "Unknown column '{}'.".format(key)
self.columns[key] = None if value == '' else value
@@ -94,9 +91,26 @@ class PlaceColumn:
if value.startswith('country:'):
ccode = value[8:].upper()
self.geometry = "ST_SetSRID(ST_Point({}, {}), 4326)".format(*ALIASES[ccode])
elif ',' not in value:
if self.grid:
pt = self.grid.parse_point(value)
else:
pt = value.split(' ')
self.geometry = f"ST_SetSRID(ST_Point({pt[0]}, {pt[1]}), 4326)"
elif '(' not in value:
if self.grid:
coords = ','.join(' '.join(f"{p:.7f}" for p in pt)
for pt in self.grid.parse_line(value))
else:
coords = value
self.geometry = f"'srid=4326;LINESTRING({coords})'::geometry"
else:
wkt = self.grid.geometry_to_wkt(value)
self.geometry = f"'srid=4326;{wkt}'::geometry"
if self.grid:
coords = ','.join(' '.join(f"{p:.7f}" for p in pt)
for pt in self.grid.parse_line(value[1:-1]))
else:
coords = value[1:-1]
self.geometry = f"'srid=4326;POLYGON(({coords}))'::geometry"
def _add_hstore(self, column, key, value):
if column in self.columns:
@@ -106,7 +120,7 @@ class PlaceColumn:
def get_wkt(self):
if self.columns['osm_type'] == 'N' and self.geometry is None:
pt = self.grid.get(str(self.columns['osm_id']))
pt = self.grid.get(str(self.columns['osm_id'])) if self.grid else None
if pt is None:
pt = (random.uniform(-180, 180), random.uniform(-90, 90))

View File

@@ -109,16 +109,14 @@ class APITester:
def add_postcode(self, **kw):
self.add_data('postcode',
{'place_id': kw.get('place_id', 1000),
'osm_id': kw.get('osm_id'),
'parent_place_id': kw.get('parent_place_id'),
'country_code': kw.get('country_code'),
'postcode': kw.get('postcode'),
'rank_search': kw.get('rank_search', 21),
'rank_search': kw.get('rank_search', 20),
'rank_address': kw.get('rank_address', 22),
'indexed_date': kw.get('indexed_date',
dt.datetime(2022, 12, 7, 14, 14, 46, 0)),
'centroid': kw.get('centroid', 'POINT(23 34)'),
'geometry': kw.get('geometry', 'POLYGON((22.99 33.99, 22.99 34.01, '
'23.01 34, 22.99 33.99))')})
'geometry': kw.get('geometry', 'POINT(23 34)')})
def add_country(self, country_code, geometry):
self.add_data('country_grid',

View File

@@ -17,9 +17,6 @@ class MyToken(query.Token):
def get_category(self):
return 'this', 'that'
def get_country(self):
return 'cc'
def mktoken(tid: int):
return MyToken(penalty=3.0, token=tid, count=1, addr_count=1,

Some files were not shown because too many files have changed in this diff Show More