Compare commits

..

1 Commits

Author SHA1 Message Date
Sarah Hoffmann
212b69799c adapt docs for release 2025-10-29 11:08:42 +01:00
159 changed files with 2391 additions and 3509 deletions

View File

@@ -1,12 +0,0 @@
## Summary
<!-- Describe the purpose of your pull request and, if present, link to existing issues. -->
## AI usage
<!-- Please list where and to what extent AI was used. -->
## Contributor guidelines (mandatory)
<!-- We only accept pull requests that follow our guidelines. A deliberate violation may result in a ban. -->
- [ ] I have adhered to the [coding style](https://github.com/osm-search/Nominatim/blob/master/CONTRIBUTING.md#coding-style)
- [ ] I have [tested](https://github.com/osm-search/Nominatim/blob/master/CONTRIBUTING.md#testing) the proposed changes
- [ ] I have [disclosed](https://github.com/osm-search/Nominatim/blob/master/CONTRIBUTING.md#using-ai-assisted-code-generators) above any use of AI to generate code, documentation, or the pull request description

View File

@@ -22,7 +22,7 @@ runs:
- name: Install prerequisites from apt - name: Install prerequisites from apt
run: | run: |
sudo apt-get install -y -qq python3-icu python3-datrie python3-jinja2 python3-psutil python3-dotenv python3-yaml python3-sqlalchemy python3-psycopg python3-asyncpg python3-mwparserfromhell sudo apt-get install -y -qq python3-icu python3-datrie python3-jinja2 python3-psutil python3-dotenv python3-yaml python3-sqlalchemy python3-psycopg python3-asyncpg
shell: bash shell: bash
if: inputs.dependencies == 'apt' if: inputs.dependencies == 'apt'

View File

@@ -1,95 +0,0 @@
name: 'Setup Postgresql and Postgis on Windows'
description: 'Installs PostgreSQL and PostGIS for Windows and configures it for CI tests'
inputs:
postgresql-version:
description: 'Version of PostgreSQL to install'
required: true
runs:
using: "composite"
steps:
- name: Set up PostgreSQL variables
shell: pwsh
run: |
$version = "${{ inputs.postgresql-version }}"
$root = "C:\Program Files\PostgreSQL\$version"
$bin = "$root\bin"
echo "PGROOT=$root" | Out-File -FilePath $env:GITHUB_ENV -Encoding utf8 -Append
echo "PGBIN=$bin" | Out-File -FilePath $env:GITHUB_ENV -Encoding utf8 -Append
echo "$bin" | Out-File -FilePath $env:GITHUB_PATH -Encoding utf8 -Append
- name: Decide Postgis version (Windows)
id: postgis-ver
shell: pwsh
run: |
echo "PowerShell version: ${PSVersionTable.PSVersion}"
$PG_VERSION = Split-Path $env:PGROOT -Leaf
$postgis_page = "https://download.osgeo.org/postgis/windows/pg$PG_VERSION"
echo "Detecting PostGIS version from $postgis_page for PostgreSQL $PG_VERSION"
$pgis_bundle = (Invoke-WebRequest -Uri $postgis_page -ErrorAction Stop).Links.Where({$_.href -match "^postgis.*zip$"}).href
if (!$pgis_bundle) {
Write-Error "Could not find latest PostGIS version in $postgis_page that would match ^postgis.*zip$ pattern"
exit 1
}
$pgis_bundle = [IO.Path]::ChangeExtension($pgis_bundle, [NullString]::Value)
$pgis_bundle_url = "$postgis_page/$pgis_bundle.zip"
Add-Content $env:GITHUB_OUTPUT "postgis_file=$pgis_bundle"
Add-Content $env:GITHUB_OUTPUT "postgis_bundle_url=$pgis_bundle_url"
- uses: actions/cache@v4
with:
path: |
C:/postgis.zip
key: postgis-cache-${{ steps.postgis-ver.outputs.postgis_file }}
- name: Download postgis
shell: pwsh
run: |
if (!(Test-Path "C:\postgis.zip")){(new-object net.webclient).DownloadFile($env:PGIS_BUNDLE_URL, "c:\postgis.zip")}
if (Test-path "c:\postgis_archive"){Remove-Item "c:\postgis_archive" -Recurse -Force}
7z x c:\postgis.zip -oc:\postgis_archive
env:
PGIS_BUNDLE_URL: ${{ steps.postgis-ver.outputs.postgis_bundle_url }}
- name: Install postgis
shell: bash
run: |
echo "Root: $PGROOT, Bin: $PGBIN"
cp -r c:/postgis_archive/postgis-bundle-*/* "$PGROOT"
- name: Start PostgreSQL on Windows
run: |
$pgService = Get-Service -Name postgresql*
Set-Service -InputObject $pgService -Status running -StartupType automatic
Start-Process -FilePath "$env:PGBIN\pg_isready" -Wait -PassThru
shell: pwsh
- name: Adapt postgresql configuration
shell: pwsh
env:
PGPASSWORD: root
run: |
& "$env:PGBIN\psql" -U postgres -d postgres -c "ALTER SYSTEM SET fsync = 'off';"
& "$env:PGBIN\psql" -U postgres -d postgres -c "ALTER SYSTEM SET synchronous_commit = 'off';"
& "$env:PGBIN\psql" -U postgres -d postgres -c "ALTER SYSTEM SET full_page_writes = 'off';"
& "$env:PGBIN\psql" -U postgres -d postgres -c "ALTER SYSTEM SET shared_buffers = '1GB';"
& "$env:PGBIN\psql" -U postgres -d postgres -c "ALTER SYSTEM SET port = 5432;"
Restart-Service -Name postgresql*
Start-Process -FilePath "$env:PGBIN\pg_isready" -Wait -PassThru
- name: Setup database users
shell: pwsh
env:
PGPASSWORD: root
run: |
& "$env:PGBIN\createuser" -U postgres -S www-data
& "$env:PGBIN\createuser" -U postgres -s runner

View File

@@ -1,7 +1,5 @@
name: 'Setup Postgresql and Postgis' name: 'Setup Postgresql and Postgis'
description: 'Installs PostgreSQL and PostGIS and configures it for CI tests'
inputs: inputs:
postgresql-version: postgresql-version:
description: 'Version of PostgreSQL to install' description: 'Version of PostgreSQL to install'

View File

@@ -140,65 +140,6 @@ jobs:
../venv/bin/python -m pytest test/bdd --nominatim-purge ../venv/bin/python -m pytest test/bdd --nominatim-purge
working-directory: Nominatim working-directory: Nominatim
tests-windows:
needs: create-archive
runs-on: windows-latest
steps:
- uses: actions/download-artifact@v4
with:
name: full-source
- name: Unpack Nominatim
run: tar xf nominatim-src.tar.bz2
- uses: ./Nominatim/.github/actions/setup-postgresql-windows
with:
postgresql-version: 17
- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: '3.14'
- name: Install Spatialite
run: |
Invoke-WebRequest -Uri "https://www.gaia-gis.it/gaia-sins/windows-bin-amd64/mod_spatialite-5.1.0-win-amd64.7z" -OutFile "spatialite.7z"
7z x spatialite.7z -o"C:\spatialite"
echo "C:\spatialite\mod_spatialite-5.1.0-win-amd64" | Out-File -FilePath $env:GITHUB_PATH -Encoding utf8 -Append
- name: Install osm2pgsql
run: |
Invoke-WebRequest -Uri "https://osm2pgsql.org/download/windows/osm2pgsql-latest-x64.zip" -OutFile "osm2pgsql.zip"
Expand-Archive -Path "osm2pgsql.zip" -DestinationPath "C:\osm2pgsql"
$BinDir = Get-ChildItem -Path "C:\osm2pgsql" -Recurse -Filter "osm2pgsql.exe" | Select-Object -ExpandProperty DirectoryName | Select-Object -First 1
if (-not $BinDir) {
Write-Error "Could not find osm2pgsql.exe"
exit 1
}
echo "$BinDir" | Out-File -FilePath $env:GITHUB_PATH -Encoding utf8 -Append
$FullExePath = Join-Path $BinDir "osm2pgsql.exe"
echo "NOMINATIM_OSM2PGSQL_BINARY=$FullExePath" | Out-File -FilePath $env:GITHUB_ENV -Encoding utf8 -Append
- name: Set UTF-8 encoding
run: |
echo "PYTHONUTF8=1" >> $env:GITHUB_ENV
[System.Console]::OutputEncoding = [System.Text.Encoding]::UTF8
- name: Install PyICU from wheel
run: |
python -m pip install https://github.com/cgohlke/pyicu-build/releases/download/v2.16.0/pyicu-2.16-cp314-cp314-win_amd64.whl
- name: Install test prerequisites
run: |
python -m pip install -U pip
python -m pip install pytest pytest-asyncio "psycopg[binary]!=3.3.0" python-dotenv pyyaml jinja2 psutil sqlalchemy pytest-bdd falcon starlette uvicorn asgi_lifespan aiosqlite osmium mwparserfromhell
- name: Python unit tests
run: |
python -m pytest test/python -k "not (import_osm or run_osm2pgsql)"
working-directory: Nominatim
install: install:
runs-on: ubuntu-latest runs-on: ubuntu-latest
needs: create-archive needs: create-archive

View File

@@ -4,7 +4,7 @@
Bugs can be reported at https://github.com/openstreetmap/Nominatim/issues. Bugs can be reported at https://github.com/openstreetmap/Nominatim/issues.
Please always open a separate issue for each problem. In particular, do Please always open a separate issue for each problem. In particular, do
not add your bugs to closed issues. They may look similar to you but not add your bugs to closed issues. They may looks similar to you but
often are completely different from the maintainer's point of view. often are completely different from the maintainer's point of view.
## Workflow for Pull Requests ## Workflow for Pull Requests
@@ -21,7 +21,7 @@ that you are responsible for your pull requests. You should be prepared
to get change requests because as the maintainers we have to make sure to get change requests because as the maintainers we have to make sure
that your contribution fits well with the rest of the code. Please make that your contribution fits well with the rest of the code. Please make
sure that you have time to react to these comments and amend the code or sure that you have time to react to these comments and amend the code or
engage in a conversation. Do not expect that others will pick up your code, engage in a conversion. Do not expect that others will pick up your code,
it will almost never happen. it will almost never happen.
Please open a separate pull request for each issue you want to address. Please open a separate pull request for each issue you want to address.
@@ -38,19 +38,10 @@ description or in documentation need to
1. clearly mark the AI-generated sections as such, for example, by 1. clearly mark the AI-generated sections as such, for example, by
mentioning all use of AI in the PR description, and mentioning all use of AI in the PR description, and
2. include proof that you have run the generated code on an actual 2. include proof that you have run the generated code on an actual
installation of Nominatim. Adding and executing tests will not be installation of Nominatim. Adding and excuting tests will not be
sufficient. You need to show that the code actually solves the problem sufficient. You need to show that the code actually solves the problem
the PR claims to solve. the PR claims to solve.
## Getting Started with Development
Please see the development section of the Nominatim documentation for
* [an architecture overview](https://nominatim.org/release-docs/develop/develop/overview/)
and backgrounds on some of the algorithms
* [how to set up a development environment](https://nominatim.org/release-docs/develop/develop/Development-Environment/)
* and background on [how tests are organised](https://nominatim.org/release-docs/develop/develop/Testing/)
## Coding style ## Coding style

View File

@@ -10,13 +10,13 @@ Nominatim. Please refer to the documentation of
[Nginx](https://nginx.org/en/docs/) for background information on how [Nginx](https://nginx.org/en/docs/) for background information on how
to configure it. to configure it.
### Installing the required packages !!! Note
Throughout this page, we assume your Nominatim project directory is
located in `/srv/nominatim-project`. If you have put it somewhere else,
you need to adjust the commands and configuration accordingly.
!!! warning
ASGI support in gunicorn requires at least version 25.0. If you need ### Installing the required packages
to work with an older version of gunicorn, please refer to
[older Nominatim deployment documentation](https://nominatim.org/release-docs/5.2/admin/Deployment-Python/)
to learn how to run gunicorn with uvicorn.
The Nominatim frontend is best run from its own virtual environment. If The Nominatim frontend is best run from its own virtual environment. If
you have already created one for the database backend during the you have already created one for the database backend during the
@@ -37,27 +37,23 @@ cd Nominatim
``` ```
The recommended way to deploy a Python ASGI application is to run The recommended way to deploy a Python ASGI application is to run
the [gunicorn](https://gunicorn.org/) HTTP server. We use the ASGI runner [uvicorn](https://www.uvicorn.org/)
together with [gunicorn](https://gunicorn.org/) HTTP server. We use
Falcon here as the web framework. Falcon here as the web framework.
Add the necessary packages to your virtual environment: Add the necessary packages to your virtual environment:
``` sh ``` sh
/srv/nominatim-venv/bin/pip install falcon gunicorn /srv/nominatim-venv/bin/pip install falcon uvicorn gunicorn
``` ```
### Setting up Nominatim as a systemd job ### Setting up Nominatim as a systemd job
!!! Note
These instructions assume your Nominatim project directory is
located in `/srv/nominatim-project`. If you have put it somewhere else,
you need to adjust the commands and configuration accordingly.
Next you need to set up the service that runs the Nominatim frontend. This is Next you need to set up the service that runs the Nominatim frontend. This is
easiest done with a systemd job. easiest done with a systemd job.
First you need to tell systemd to create a socket file to be used by First you need to tell systemd to create a socket file to be used by
gunicorn. Create the following file `/etc/systemd/system/nominatim.socket`: hunicorn. Create the following file `/etc/systemd/system/nominatim.socket`:
``` systemd ``` systemd
[Unit] [Unit]
@@ -85,8 +81,10 @@ Type=simple
User=www-data User=www-data
Group=www-data Group=www-data
WorkingDirectory=/srv/nominatim-project WorkingDirectory=/srv/nominatim-project
ExecStart=/srv/nominatim-venv/bin/gunicorn -b unix:/run/nominatim.sock -w 4 --worker-class asgi --protocol uwsgi --worker-connections 1000 "nominatim_api.server.falcon.server:run_wsgi()" ExecStart=/srv/nominatim-venv/bin/gunicorn -b unix:/run/nominatim.sock -w 4 -k uvicorn.workers.UvicornWorker "nominatim_api.server.falcon.server:run_wsgi()"
ExecReload=/bin/kill -s HUP $MAINPID ExecReload=/bin/kill -s HUP $MAINPID
StandardOutput=append:/var/log/gunicorn-nominatim.log
StandardError=inherit
PrivateTmp=true PrivateTmp=true
TimeoutStopSec=5 TimeoutStopSec=5
KillMode=mixed KillMode=mixed
@@ -98,10 +96,7 @@ WantedBy=multi-user.target
This sets up gunicorn with 4 workers (`-w 4` in ExecStart). Each worker runs This sets up gunicorn with 4 workers (`-w 4` in ExecStart). Each worker runs
its own Python process using its own Python process using
[`NOMINATIM_API_POOL_SIZE`](../customize/Settings.md#nominatim_api_pool_size) [`NOMINATIM_API_POOL_SIZE`](../customize/Settings.md#nominatim_api_pool_size)
connections to the database to serve requests in parallel. The parameter connections to the database to serve requests in parallel.
`--worker-connections` restricts how many requests gunicorn will queue for
each worker. This can help distribute work better when the server is under
high load.
Make the new services known to systemd and start it: Make the new services known to systemd and start it:
@@ -113,15 +108,13 @@ sudo systemctl enable nominatim.service
sudo systemctl start nominatim.service sudo systemctl start nominatim.service
``` ```
This sets the service up so that Nominatim is automatically started This sets the service up, so that Nominatim is automatically started
on reboot. on reboot.
### Configuring nginx ### Configuring nginx
To make the service available to the world, you need to proxy it through To make the service available to the world, you need to proxy it through
nginx. We use the binary uwsgi protocol to speed up communication nginx. Add the following definition to the default configuration:
between nginx and gunicorn. Add the following definition to the default
configuration:
``` nginx ``` nginx
upstream nominatim_service { upstream nominatim_service {
@@ -136,8 +129,11 @@ server {
index /search; index /search;
location / { location / {
uwsgi_pass nominatim_service; proxy_set_header Host $http_host;
include uwsgi_params; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_redirect off;
proxy_pass http://nominatim_service;
} }
} }
``` ```

View File

@@ -37,7 +37,6 @@ Furthermore the following Python libraries are required:
* [Jinja2](https://palletsprojects.com/p/jinja/) * [Jinja2](https://palletsprojects.com/p/jinja/)
* [PyICU](https://pypi.org/project/PyICU/) * [PyICU](https://pypi.org/project/PyICU/)
* [PyYaml](https://pyyaml.org/) (5.1+) * [PyYaml](https://pyyaml.org/) (5.1+)
* [mwparserfromhell](https://github.com/earwig/mwparserfromhell/)
These will be installed automatically when using pip installation. These will be installed automatically when using pip installation.
@@ -111,14 +110,17 @@ Then you can install Nominatim with:
pip install nominatim-db nominatim-api pip install nominatim-db nominatim-api
## Downloading and building Nominatim ## Downloading and building Nominatim from source
The following instructions are only relevant, if you want to build and
install Nominatim **from source**.
### Downloading the latest release ### Downloading the latest release
You can download the [latest release from nominatim.org](https://nominatim.org/downloads/). You can download the [latest release from nominatim.org](https://nominatim.org/downloads/).
The release contains all necessary files. Just unpack it. The release contains all necessary files. Just unpack it.
### Downloading the latest development version ### Downloading the source for the latest development version
If you want to install latest development version from github: If you want to install latest development version from github:
@@ -132,7 +134,7 @@ The development version does not include the country grid. Download it separatel
wget -O Nominatim/data/country_osm_grid.sql.gz https://nominatim.org/data/country_grid.sql.gz wget -O Nominatim/data/country_osm_grid.sql.gz https://nominatim.org/data/country_grid.sql.gz
``` ```
### Building Nominatim ### Building Nominatim from source
Nominatim is easiest to run from its own virtual environment. To create one, run: Nominatim is easiest to run from its own virtual environment. To create one, run:

View File

@@ -113,7 +113,6 @@ The following classifications are recognized:
| named | Consider as main tag, when the object has a primary name (see [names](#name-tags) below) | | named | Consider as main tag, when the object has a primary name (see [names](#name-tags) below) |
| named_with_key | Consider as main tag, when the object has a primary name with a domain prefix. For example, if the main tag is `bridge=yes`, then it will only be added as an extra entry, if there is a tag `bridge:name[:XXX]` for the same object. If this property is set, all names that are not domain-specific are ignored. | | named_with_key | Consider as main tag, when the object has a primary name with a domain prefix. For example, if the main tag is `bridge=yes`, then it will only be added as an extra entry, if there is a tag `bridge:name[:XXX]` for the same object. If this property is set, all names that are not domain-specific are ignored. |
| fallback | Consider as main tag only when no other main tag was found. Fallback always implies `named`, i.e. fallbacks are only tried for objects with primary names. | | fallback | Consider as main tag only when no other main tag was found. Fallback always implies `named`, i.e. fallbacks are only tried for objects with primary names. |
| postcode_area | Tag indicates a postcode area. Copy area into the table of postcodes but only when the object is a relation and has a postcode tagged. |
| delete | Completely ignore the tag in any further processing | | delete | Completely ignore the tag in any further processing |
| extra | Move the tag to extratags and then ignore it for further processing | | extra | Move the tag to extratags and then ignore it for further processing |
| `<function>`| Advanced handling, see [below](#advanced-main-tag-handling) | | `<function>`| Advanced handling, see [below](#advanced-main-tag-handling) |

View File

@@ -229,7 +229,7 @@ _None._
| Option | Description | | Option | Description |
|-----------------|-------------| |-----------------|-------------|
| locales | [Locales](../library/Result-Handling.md#locale) object for the requested language(s) | | locales | [Locale](../library/Result-Handling.md#locale) object for the requested language(s) |
| group_hierarchy | Setting of [group_hierarchy](../api/Details.md#output-details) parameter | | group_hierarchy | Setting of [group_hierarchy](../api/Details.md#output-details) parameter |
| icon_base_url | (optional) URL pointing to icons as set in [NOMINATIM_MAPICON_URL](Settings.md#nominatim_mapicon_url) | | icon_base_url | (optional) URL pointing to icons as set in [NOMINATIM_MAPICON_URL](Settings.md#nominatim_mapicon_url) |

View File

@@ -79,7 +79,7 @@ the placex table. Only three columns are special:
Address interpolations are always ways in OSM, which is why there is no column Address interpolations are always ways in OSM, which is why there is no column
`osm_type`. `osm_type`.
The **location_postcodes** table holds computed centroids of all postcodes that The **location_postcode** table holds computed centroids of all postcodes that
can be found in the OSM data. The meaning of the columns is again the same can be found in the OSM data. The meaning of the columns is again the same
as that of the placex table. as that of the placex table.

View File

@@ -56,7 +56,7 @@ The easiest way, to handle these Python dependencies is to run your
development from within a virtual environment. development from within a virtual environment.
```sh ```sh
sudo apt install build-essential libsqlite3-mod-spatialite osm2pgsql \ sudo apt install libsqlite3-mod-spatialite osm2pgsql \
postgresql-postgis postgresql-postgis-scripts \ postgresql-postgis postgresql-postgis-scripts \
pkg-config libicu-dev virtualenv pkg-config libicu-dev virtualenv
``` ```
@@ -68,12 +68,12 @@ virtualenv ~/nominatim-dev-venv
~/nominatim-dev-venv/bin/pip install\ ~/nominatim-dev-venv/bin/pip install\
psutil 'psycopg[binary]' PyICU SQLAlchemy \ psutil 'psycopg[binary]' PyICU SQLAlchemy \
python-dotenv jinja2 pyYAML \ python-dotenv jinja2 pyYAML \
mkdocs 'mkdocstrings[python]' mkdocs-gen-files mkdocs-material \ mkdocs 'mkdocstrings[python]' mkdocs-gen-files \
pytest pytest-asyncio pytest-bdd flake8 \ pytest pytest-asyncio pytest-bdd flake8 \
types-jinja2 types-markupsafe types-psutil types-psycopg2 \ types-jinja2 types-markupsafe types-psutil types-psycopg2 \
types-pygments types-pyyaml types-requests types-ujson \ types-pygments types-pyyaml types-requests types-ujson \
types-urllib3 typing-extensions gunicorn falcon starlette \ types-urllib3 typing-extensions unicorn falcon starlette \
uvicorn mypy osmium aiosqlite mwparserfromhell uvicorn mypy osmium aiosqlite
``` ```
Now enter the virtual environment whenever you want to develop: Now enter the virtual environment whenever you want to develop:

View File

@@ -52,15 +52,6 @@ To run the functional tests, do
pytest test/bdd pytest test/bdd
You can run a single feature file using expression matching:
pytest test/bdd -k osm2pgsql/import/entrances.feature
This even works for running single tests by adding the line number of the
scenario header like that:
pytest test/bdd -k 'osm2pgsql/import/entrances.feature and L4'
The BDD tests create databases for the tests. You can set name of the databases The BDD tests create databases for the tests. You can set name of the databases
through configuration variables in your `pytest.ini`: through configuration variables in your `pytest.ini`:

View File

@@ -74,16 +74,15 @@ map place_addressline {
isaddress => BOOLEAN isaddress => BOOLEAN
} }
map location_postcodes { map location_postcode {
place_id => BIGINT place_id => BIGINT
osm_id => BIGINT
postcode => TEXT postcode => TEXT
parent_place_id => BIGINT parent_place_id => BIGINT
rank_search => SMALLINT rank_search => SMALLINT
rank_address => SMALLINT
indexed_status => SMALLINT indexed_status => SMALLINT
indexed_date => TIMESTAMP indexed_date => TIMESTAMP
geometry => GEOMETRY geometry => GEOMETRY
centroid -> GEOMETRY
} }
placex::place_id <-- search_name::place_id placex::place_id <-- search_name::place_id
@@ -95,6 +94,6 @@ search_name::nameaddress_vector --> word::word_id
place_addressline -[hidden]> location_property_osmline place_addressline -[hidden]> location_property_osmline
search_name -[hidden]> place_addressline search_name -[hidden]> place_addressline
location_property_osmline -[hidden]-> location_postcodes location_property_osmline -[hidden]-> location_postcode
@enduml @enduml

View File

@@ -248,19 +248,19 @@ of the result. To do that, you first need to decide in which language the
results should be presented. As with the names in the result itself, the results should be presented. As with the names in the result itself, the
places in `address_rows` contain all possible name translation for each row. places in `address_rows` contain all possible name translation for each row.
The library has a helper class `Locales` which helps extracting a name of a The library has a helper class `Locale` which helps extracting a name of a
place in the preferred language. It takes a single parameter with a list place in the preferred language. It takes a single parameter with a list
of language codes in the order of preference. So of language codes in the order of preference. So
``` python ``` python
locale = napi.Locales(['fr', 'en']) locale = napi.Locale(['fr', 'en'])
``` ```
creates a helper class that returns the name preferably in French. If that is creates a helper class that returns the name preferably in French. If that is
not possible, it tries English and eventually falls back to the default `name` not possible, it tries English and eventually falls back to the default `name`
or `ref`. or `ref`.
The `Locales` object can be applied to a name dictionary to return the best-matching The `Locale` object can be applied to a name dictionary to return the best-matching
name out of it: name out of it:
``` python ``` python
@@ -273,7 +273,7 @@ component based on its `local_name` field. This is then utilized by the overall
which has a helper function to apply the function to all its address_row members and saves which has a helper function to apply the function to all its address_row members and saves
the result in the `locale_name` field. the result in the `locale_name` field.
However, in order to set this `local_name` field in a preferred language, you must use the `Locales` However, in order to set this `local_name` field in a preferred language, you must use the `Locale`
object which contains the function `localize_results`, which explicitly sets each `local_name field`. object which contains the function `localize_results`, which explicitly sets each `local_name field`.
``` python ``` python

View File

@@ -13,8 +13,7 @@ for infile in VAGRANT_PATH.glob('Install-on-*.sh'):
outfile = f"admin/{infile.stem}.md" outfile = f"admin/{infile.stem}.md"
title = infile.stem.replace('-', ' ') title = infile.stem.replace('-', ' ')
with mkdocs_gen_files.open(outfile, "w", encoding='utf-8') as outfd, \ with mkdocs_gen_files.open(outfile, "w") as outfd, infile.open() as infd:
infile.open(encoding='utf-8') as infd:
print("#", title, file=outfd) print("#", title, file=outfd)
has_empty = False has_empty = False
for line in infd: for line in infd:

View File

@@ -65,18 +65,6 @@ local table_definitions = {
{ column = 'geometry', type = 'geometry', projection = 'WGS84', not_null = true } { column = 'geometry', type = 'geometry', projection = 'WGS84', not_null = true }
}, },
indexes = {} indexes = {}
},
place_postcode = {
ids = { type = 'any', id_column = 'osm_id', type_column = 'osm_type' },
columns = {
{ column = 'postcode', type = 'text', not_null = true },
{ column = 'country_code', type = 'text' },
{ column = 'centroid', type = 'point', projection = 'WGS84', not_null = true },
{ column = 'geometry', type = 'geometry', projection = 'WGS84' }
},
indexes = {
{ column = 'postcode', method = 'btree' }
}
} }
} }
@@ -125,7 +113,6 @@ local PlaceTransform = {}
-- Special transform meanings which are interpreted elsewhere -- Special transform meanings which are interpreted elsewhere
PlaceTransform.fallback = 'fallback' PlaceTransform.fallback = 'fallback'
PlaceTransform.postcode_area = 'postcode_area'
PlaceTransform.delete = 'delete' PlaceTransform.delete = 'delete'
PlaceTransform.extra = 'extra' PlaceTransform.extra = 'extra'
@@ -432,25 +419,11 @@ function Place:write_place(k, v, mfunc)
return 0 return 0
end end
function Place:write_row(k, v)
function Place:geometry_is_valid()
if self.geometry == nil then if self.geometry == nil then
self.geometry = self.geom_func(self.object) self.geometry = self.geom_func(self.object)
end
if self.geometry == nil or self.geometry:is_null() then if self.geometry == nil or self.geometry:is_null() then
self.geometry = false
return false
end
return true
end
return self.geometry ~= false
end
function Place:write_row(k, v)
if not self:geometry_is_valid() then
return 0 return 0
end end
@@ -702,6 +675,9 @@ function module.process_tags(o)
if o.address.country ~= nil and #o.address.country ~= 2 then if o.address.country ~= nil and #o.address.country ~= 2 then
o.address['country'] = nil o.address['country'] = nil
end end
if POSTCODE_FALLBACK and fallback == nil and o.address.postcode ~= nil then
fallback = {'place', 'postcode', PlaceTransform.always}
end
if o.address.interpolation ~= nil then if o.address.interpolation ~= nil then
o:write_place('place', 'houses', PlaceTransform.always) o:write_place('place', 'houses', PlaceTransform.always)
@@ -709,41 +685,20 @@ function module.process_tags(o)
end end
-- collect main keys -- collect main keys
local postcode_collect = false
for k, v in pairs(o.intags) do for k, v in pairs(o.intags) do
local ktable = MAIN_KEYS[k] local ktable = MAIN_KEYS[k]
if ktable then if ktable then
local ktype = ktable[v] or ktable[1] local ktype = ktable[v] or ktable[1]
if type(ktype) == 'function' then if type(ktype) == 'function' then
o:write_place(k, v, ktype) o:write_place(k, v, ktype)
elseif ktype == 'postcode_area' then
postcode_collect = true
if o.object.type == 'relation'
and o.address.postcode ~= nil
and o:geometry_is_valid() then
insert_row.place_postcode{
postcode = o.address.postcode,
centroid = o.geometry:centroid(),
geometry = o.geometry
}
end
elseif ktype == 'fallback' and o.has_name then elseif ktype == 'fallback' and o.has_name then
fallback = {k, v, PlaceTransform.named} fallback = {k, v, PlaceTransform.named}
end end
end end
end end
if o.num_entries == 0 then if fallback ~= nil and o.num_entries == 0 then
if fallback ~= nil then
o:write_place(fallback[1], fallback[2], fallback[3]) o:write_place(fallback[1], fallback[2], fallback[3])
elseif POSTCODE_FALLBACK and not postcode_collect
and o.address.postcode ~= nil
and o:geometry_is_valid() then
insert_row.place_postcode{
postcode = o.address.postcode,
centroid = o.geometry:centroid()
}
end
end end
end end

View File

@@ -117,8 +117,7 @@ module.MAIN_TAGS.all_boundaries = {
boundary = {'named', boundary = {'named',
place = 'delete', place = 'delete',
land_area = 'delete', land_area = 'delete',
protected_area = 'fallback', postal_code = 'always'},
postal_code = 'postcode_area'},
landuse = 'fallback', landuse = 'fallback',
place = 'always' place = 'always'
} }
@@ -199,7 +198,7 @@ module.MAIN_TAGS_POIS = function (group)
no = group}, no = group},
landuse = {cemetery = 'always'}, landuse = {cemetery = 'always'},
leisure = {'always', leisure = {'always',
nature_reserve = 'named', nature_reserve = 'fallback',
swimming_pool = 'named', swimming_pool = 'named',
garden = 'named', garden = 'named',
common = 'named', common = 'named',
@@ -322,6 +321,7 @@ module.NAME_TAGS = {}
module.NAME_TAGS.core = {main = {'name', 'name:*', module.NAME_TAGS.core = {main = {'name', 'name:*',
'int_name', 'int_name:*', 'int_name', 'int_name:*',
'nat_name', 'nat_name:*',
'reg_name', 'reg_name:*', 'reg_name', 'reg_name:*',
'loc_name', 'loc_name:*', 'loc_name', 'loc_name:*',
'old_name', 'old_name:*', 'old_name', 'old_name:*',

View File

@@ -2,7 +2,7 @@
-- --
-- This file is part of Nominatim. (https://nominatim.org) -- This file is part of Nominatim. (https://nominatim.org)
-- --
-- Copyright (C) 2025 by the Nominatim developer community. -- Copyright (C) 2022 by the Nominatim developer community.
-- For a full list of authors see the git log. -- For a full list of authors see the git log.
{% include('functions/utils.sql') %} {% include('functions/utils.sql') %}
@@ -18,7 +18,7 @@
{% include 'functions/placex_triggers.sql' %} {% include 'functions/placex_triggers.sql' %}
{% endif %} {% endif %}
{% if 'location_postcodes' in db.tables %} {% if 'location_postcode' in db.tables %}
{% include 'functions/postcode_triggers.sql' %} {% include 'functions/postcode_triggers.sql' %}
{% endif %} {% endif %}

View File

@@ -2,7 +2,7 @@
-- --
-- This file is part of Nominatim. (https://nominatim.org) -- This file is part of Nominatim. (https://nominatim.org)
-- --
-- Copyright (C) 2026 by the Nominatim developer community. -- Copyright (C) 2022 by the Nominatim developer community.
-- For a full list of authors see the git log. -- For a full list of authors see the git log.
-- Functions for interpreting wkipedia/wikidata tags and computing importance. -- Functions for interpreting wkipedia/wikidata tags and computing importance.
@@ -166,7 +166,7 @@ BEGIN
END LOOP; END LOOP;
-- Nothing? Then try with the wikidata tag. -- Nothing? Then try with the wikidata tag.
IF extratags ? 'wikidata' THEN IF result.importance is null AND extratags ? 'wikidata' THEN
FOR match IN FOR match IN
{% if 'wikimedia_importance' in db.tables %} {% if 'wikimedia_importance' in db.tables %}
SELECT * FROM wikimedia_importance SELECT * FROM wikimedia_importance
@@ -185,7 +185,9 @@ BEGIN
END IF; END IF;
-- Still nothing? Fall back to a default. -- Still nothing? Fall back to a default.
IF result.importance is null THEN
result.importance := 0.40001 - (rank_search::float / 75); result.importance := 0.40001 - (rank_search::float / 75);
END IF;
{% if 'secondary_importance' in db.tables %} {% if 'secondary_importance' in db.tables %}
FOR match IN FOR match IN
@@ -193,10 +195,8 @@ BEGIN
FROM secondary_importance FROM secondary_importance
WHERE ST_Intersects(ST_ConvexHull(rast), centroid) LIMIT 1 WHERE ST_Intersects(ST_ConvexHull(rast), centroid) LIMIT 1
LOOP LOOP
IF match.importance is not NULL THEN
-- Secondary importance as tie breaker with 0.0001 weight. -- Secondary importance as tie breaker with 0.0001 weight.
result.importance := result.importance + match.importance::float / 655350000; result.importance := result.importance + match.importance::float / 655350000;
END IF;
END LOOP; END LOOP;
{% endif %} {% endif %}

View File

@@ -2,7 +2,7 @@
-- --
-- This file is part of Nominatim. (https://nominatim.org) -- This file is part of Nominatim. (https://nominatim.org)
-- --
-- Copyright (C) 2026 by the Nominatim developer community. -- Copyright (C) 2022 by the Nominatim developer community.
-- For a full list of authors see the git log. -- For a full list of authors see the git log.
DROP TYPE IF EXISTS nearfeaturecentr CASCADE; DROP TYPE IF EXISTS nearfeaturecentr CASCADE;
@@ -123,12 +123,10 @@ BEGIN
RETURN TRUE; RETURN TRUE;
END IF; END IF;
IF in_rank_search <= 4 THEN IF in_rank_search <= 4 and not in_estimate THEN
IF not in_estimate and in_country_code is not NULL THEN
INSERT INTO location_area_country (place_id, country_code, geometry) INSERT INTO location_area_country (place_id, country_code, geometry)
(SELECT in_place_id, in_country_code, geom (SELECT in_place_id, in_country_code, geom
FROM split_geometry(in_geometry) as geom); FROM split_geometry(in_geometry) as geom);
END IF;
RETURN TRUE; RETURN TRUE;
END IF; END IF;
@@ -214,6 +212,7 @@ DECLARE
BEGIN BEGIN
{% for partition in db.partitions %} {% for partition in db.partitions %}
IF in_partition = {{ partition }} THEN IF in_partition = {{ partition }} THEN
DELETE FROM search_name_{{ partition }} values WHERE place_id = in_place_id;
IF in_rank_address > 0 THEN IF in_rank_address > 0 THEN
INSERT INTO search_name_{{ partition }} (place_id, address_rank, name_vector, centroid) INSERT INTO search_name_{{ partition }} (place_id, address_rank, name_vector, centroid)
values (in_place_id, in_rank_address, in_name_vector, in_geometry); values (in_place_id, in_rank_address, in_name_vector, in_geometry);
@@ -252,6 +251,7 @@ BEGIN
{% for partition in db.partitions %} {% for partition in db.partitions %}
IF in_partition = {{ partition }} THEN IF in_partition = {{ partition }} THEN
DELETE FROM location_road_{{ partition }} where place_id = in_place_id;
INSERT INTO location_road_{{ partition }} (partition, place_id, country_code, geometry) INSERT INTO location_road_{{ partition }} (partition, place_id, country_code, geometry)
values (in_partition, in_place_id, in_country_code, in_geometry); values (in_partition, in_place_id, in_country_code, in_geometry);
RETURN TRUE; RETURN TRUE;

View File

@@ -2,7 +2,7 @@
-- --
-- This file is part of Nominatim. (https://nominatim.org) -- This file is part of Nominatim. (https://nominatim.org)
-- --
-- Copyright (C) 2026 by the Nominatim developer community. -- Copyright (C) 2022 by the Nominatim developer community.
-- For a full list of authors see the git log. -- For a full list of authors see the git log.
CREATE OR REPLACE FUNCTION place_insert() CREATE OR REPLACE FUNCTION place_insert()
@@ -66,8 +66,7 @@ BEGIN
-- They get their parent from the interpolation. -- They get their parent from the interpolation.
UPDATE placex p SET indexed_status = 2 UPDATE placex p SET indexed_status = 2
FROM planet_osm_ways w FROM planet_osm_ways w
WHERE w.id = NEW.osm_id and p.osm_type = 'N' and p.osm_id = any(w.nodes) WHERE w.id = NEW.osm_id and p.osm_type = 'N' and p.osm_id = any(w.nodes);
and indexed_status = 0;
-- If there is already an entry in place, just update that, if necessary. -- If there is already an entry in place, just update that, if necessary.
IF existing.osm_type is not null THEN IF existing.osm_type is not null THEN
@@ -90,6 +89,35 @@ BEGIN
RETURN NEW; RETURN NEW;
END IF; END IF;
-- ---- Postcode points.
IF NEW.class = 'place' AND NEW.type = 'postcode' THEN
-- Pure postcodes are never queried from placex so we don't add them.
-- location_postcodes is filled from the place table directly.
-- Remove any old placex entry.
DELETE FROM placex WHERE osm_type = NEW.osm_type and osm_id = NEW.osm_id;
IF existing.osm_type IS NOT NULL THEN
IF coalesce(existing.address, ''::hstore) != coalesce(NEW.address, ''::hstore)
OR existing.geometry::text != NEW.geometry::text
THEN
UPDATE place
SET name = NEW.name,
address = NEW.address,
extratags = NEW.extratags,
admin_level = NEW.admin_level,
geometry = NEW.geometry
WHERE osm_type = NEW.osm_type and osm_id = NEW.osm_id
and class = NEW.class and type = NEW.type;
END IF;
RETURN NULL;
END IF;
RETURN NEW;
END IF;
-- ---- All other place types. -- ---- All other place types.
-- When an area is changed from large to small: log and discard change -- When an area is changed from large to small: log and discard change
@@ -241,6 +269,17 @@ BEGIN
WHERE osm_type = NEW.osm_type and osm_id = NEW.osm_id WHERE osm_type = NEW.osm_type and osm_id = NEW.osm_id
and class = NEW.class and type = NEW.type; and class = NEW.class and type = NEW.type;
-- Postcode areas are only kept, when there is an actual postcode assigned.
IF NEW.class = 'boundary' AND NEW.type = 'postal_code' THEN
IF NEW.address is NULL OR NOT NEW.address ? 'postcode' THEN
-- postcode was deleted, no longer retain in placex
DELETE FROM placex where place_id = existingplacex.place_id;
RETURN NULL;
END IF;
NEW.name := hstore('ref', NEW.address->'postcode');
END IF;
-- Boundaries must be areas. -- Boundaries must be areas.
IF NEW.class in ('boundary') IF NEW.class in ('boundary')
AND ST_GeometryType(NEW.geometry) not in ('ST_Polygon','ST_MultiPolygon') AND ST_GeometryType(NEW.geometry) not in ('ST_Polygon','ST_MultiPolygon')

View File

@@ -2,7 +2,7 @@
-- --
-- This file is part of Nominatim. (https://nominatim.org) -- This file is part of Nominatim. (https://nominatim.org)
-- --
-- Copyright (C) 2026 by the Nominatim developer community. -- Copyright (C) 2025 by the Nominatim developer community.
-- For a full list of authors see the git log. -- For a full list of authors see the git log.
-- Trigger functions for the placex table. -- Trigger functions for the placex table.
@@ -29,7 +29,6 @@ DECLARE
location RECORD; location RECORD;
result prepare_update_info; result prepare_update_info;
extra_names HSTORE; extra_names HSTORE;
default_language VARCHAR(10);
BEGIN BEGIN
IF not p.address ? '_inherited' THEN IF not p.address ? '_inherited' THEN
result.address := p.address; result.address := p.address;
@@ -86,13 +85,6 @@ BEGIN
IF location.name is not NULL THEN IF location.name is not NULL THEN
{% if debug %}RAISE WARNING 'Names original: %, location: %', result.name, location.name;{% endif %} {% if debug %}RAISE WARNING 'Names original: %, location: %', result.name, location.name;{% endif %}
-- Add the linked-place (e.g. city) name as a searchable placename in the default language (if any)
default_language := get_country_language_code(location.country_code);
IF default_language is not NULL AND location.name ? 'name' AND NOT location.name ? ('name:' || default_language) THEN
location.name := location.name || hstore('name:' || default_language, location.name->'name');
END IF;
-- Add all names from the place nodes that deviate from the name -- Add all names from the place nodes that deviate from the name
-- in the relation with the prefix '_place_'. Deviation means that -- in the relation with the prefix '_place_'. Deviation means that
-- either the value is different or a given key is missing completely -- either the value is different or a given key is missing completely
@@ -312,6 +304,7 @@ DECLARE
BEGIN BEGIN
IF bnd.rank_search >= 26 or bnd.rank_address = 0 IF bnd.rank_search >= 26 or bnd.rank_address = 0
or ST_GeometryType(bnd.geometry) NOT IN ('ST_Polygon','ST_MultiPolygon') or ST_GeometryType(bnd.geometry) NOT IN ('ST_Polygon','ST_MultiPolygon')
or bnd.type IN ('postcode', 'postal_code')
THEN THEN
RETURN NULL; RETURN NULL;
END IF; END IF;
@@ -348,25 +341,10 @@ BEGIN
END IF; END IF;
END IF; END IF;
IF bnd.extratags ? 'wikidata' THEN
FOR linked_placex IN
SELECT * FROM placex
WHERE placex.class = 'place' AND placex.osm_type = 'N'
AND placex.extratags ? 'wikidata' -- needed to select right index
AND placex.extratags->'wikidata' = bnd.extratags->'wikidata'
AND (placex.linked_place_id is null or placex.linked_place_id = bnd.place_id)
AND placex.rank_search < 26
AND _st_covers(bnd.geometry, placex.geometry)
ORDER BY lower(name->'name') = bnd_name desc
LOOP
{% if debug %}RAISE WARNING 'Found wikidata-matching place node %', linked_placex.osm_id;{% endif %}
RETURN linked_placex;
END LOOP;
END IF;
-- If extratags has a place tag, look for linked nodes by their place type. -- If extratags has a place tag, look for linked nodes by their place type.
-- Area and node still have to have the same name. -- Area and node still have to have the same name.
IF bnd.extratags ? 'place' and bnd_name is not null IF bnd.extratags ? 'place' and bnd.extratags->'place' != 'postcode'
and bnd_name is not null
THEN THEN
FOR linked_placex IN FOR linked_placex IN
SELECT * FROM placex SELECT * FROM placex
@@ -383,6 +361,22 @@ BEGIN
END LOOP; END LOOP;
END IF; END IF;
IF bnd.extratags ? 'wikidata' THEN
FOR linked_placex IN
SELECT * FROM placex
WHERE placex.class = 'place' AND placex.osm_type = 'N'
AND placex.extratags ? 'wikidata' -- needed to select right index
AND placex.extratags->'wikidata' = bnd.extratags->'wikidata'
AND (placex.linked_place_id is null or placex.linked_place_id = bnd.place_id)
AND placex.rank_search < 26
AND _st_covers(bnd.geometry, placex.geometry)
ORDER BY lower(name->'name') = bnd_name desc
LOOP
{% if debug %}RAISE WARNING 'Found wikidata-matching place node %', linked_placex.osm_id;{% endif %}
RETURN linked_placex;
END LOOP;
END IF;
-- Name searches can be done for ways as well as relations -- Name searches can be done for ways as well as relations
IF bnd_name is not null THEN IF bnd_name is not null THEN
{% if debug %}RAISE WARNING 'Looking for nodes with matching names';{% endif %} {% if debug %}RAISE WARNING 'Looking for nodes with matching names';{% endif %}
@@ -399,6 +393,7 @@ BEGIN
AND placex.class = 'place' AND placex.class = 'place'
AND (placex.linked_place_id is null or placex.linked_place_id = bnd.place_id) AND (placex.linked_place_id is null or placex.linked_place_id = bnd.place_id)
AND placex.rank_search < 26 -- needed to select the right index AND placex.rank_search < 26 -- needed to select the right index
AND placex.type != 'postcode'
AND ST_Covers(bnd.geometry, placex.geometry) AND ST_Covers(bnd.geometry, placex.geometry)
LOOP LOOP
{% if debug %}RAISE WARNING 'Found matching place node %', linked_placex.osm_id;{% endif %} {% if debug %}RAISE WARNING 'Found matching place node %', linked_placex.osm_id;{% endif %}
@@ -473,7 +468,7 @@ BEGIN
END IF; END IF;
END LOOP; END LOOP;
name_vector := COALESCE(token_get_name_search_tokens(token_info), '{}'::INTEGER[]); name_vector := token_get_name_search_tokens(token_info);
-- Check if the parent covers all address terms. -- Check if the parent covers all address terms.
-- If not, create a search name entry with the house number as the name. -- If not, create a search name entry with the house number as the name.
@@ -680,7 +675,7 @@ CREATE OR REPLACE FUNCTION placex_insert()
AS $$ AS $$
DECLARE DECLARE
postcode TEXT; postcode TEXT;
result INT; result BOOLEAN;
is_area BOOLEAN; is_area BOOLEAN;
country_code VARCHAR(2); country_code VARCHAR(2);
diameter FLOAT; diameter FLOAT;
@@ -702,7 +697,17 @@ BEGIN
ELSE ELSE
is_area := ST_GeometryType(NEW.geometry) IN ('ST_Polygon','ST_MultiPolygon'); is_area := ST_GeometryType(NEW.geometry) IN ('ST_Polygon','ST_MultiPolygon');
IF NEW.class = 'highway' AND is_area AND NEW.name is null IF NEW.class in ('place','boundary')
AND NEW.type in ('postcode','postal_code')
THEN
IF NEW.address IS NULL OR NOT NEW.address ? 'postcode' THEN
-- most likely just a part of a multipolygon postcode boundary, throw it away
RETURN NULL;
END IF;
NEW.name := hstore('ref', NEW.address->'postcode');
ELSEIF NEW.class = 'highway' AND is_area AND NEW.name is null
AND NEW.extratags ? 'area' AND NEW.extratags->'area' = 'yes' AND NEW.extratags ? 'area' AND NEW.extratags->'area' = 'yes'
THEN THEN
RETURN NULL; RETURN NULL;
@@ -785,12 +790,11 @@ BEGIN
-- add to tables for special search -- add to tables for special search
-- Note: won't work on initial import because the classtype tables
-- do not yet exist. It won't hurt either.
classtable := 'place_classtype_' || NEW.class || '_' || NEW.type; classtable := 'place_classtype_' || NEW.class || '_' || NEW.type;
SELECT count(*) INTO result SELECT count(*)>0 FROM pg_tables WHERE tablename = classtable and schemaname = current_schema() INTO result;
FROM pg_tables IF result THEN
WHERE classtable NOT SIMILAR TO '%\W%'
AND tablename = classtable and schemaname = current_schema();
IF result > 0 THEN
EXECUTE 'INSERT INTO ' || classtable::regclass || ' (place_id, centroid) VALUES ($1,$2)' EXECUTE 'INSERT INTO ' || classtable::regclass || ' (place_id, centroid) VALUES ($1,$2)'
USING NEW.place_id, NEW.centroid; USING NEW.place_id, NEW.centroid;
END IF; END IF;
@@ -849,7 +853,6 @@ BEGIN
NEW.indexed_date = now(); NEW.indexed_date = now();
IF OLD.indexed_status > 1 THEN
{% if 'search_name' in db.tables %} {% if 'search_name' in db.tables %}
DELETE from search_name WHERE place_id = NEW.place_id; DELETE from search_name WHERE place_id = NEW.place_id;
{% endif %} {% endif %}
@@ -857,7 +860,6 @@ BEGIN
DELETE FROM place_addressline WHERE place_id = NEW.place_id; DELETE FROM place_addressline WHERE place_id = NEW.place_id;
result := deleteRoad(NEW.partition, NEW.place_id); result := deleteRoad(NEW.partition, NEW.place_id);
result := deleteLocationArea(NEW.partition, NEW.place_id, NEW.rank_search); result := deleteLocationArea(NEW.partition, NEW.place_id, NEW.rank_search);
END IF;
NEW.extratags := NEW.extratags - 'linked_place'::TEXT; NEW.extratags := NEW.extratags - 'linked_place'::TEXT;
IF NEW.extratags = ''::hstore THEN IF NEW.extratags = ''::hstore THEN
@@ -870,12 +872,19 @@ BEGIN
NEW.linked_place_id := OLD.linked_place_id; NEW.linked_place_id := OLD.linked_place_id;
-- Remove linkage, if we have computed a different new linkee. -- Remove linkage, if we have computed a different new linkee.
IF OLD.indexed_status > 1 THEN UPDATE placex SET linked_place_id = null, indexed_status = 2
UPDATE placex
SET linked_place_id = null,
indexed_status = CASE WHEN indexed_status = 0 THEN 2 ELSE indexed_status END
WHERE linked_place_id = NEW.place_id WHERE linked_place_id = NEW.place_id
and (linked_place is null or place_id != linked_place); and (linked_place is null or linked_place_id != linked_place);
-- update not necessary for osmline, cause linked_place_id does not exist
-- Postcodes are just here to compute the centroids. They are not searchable
-- unless they are a boundary=postal_code.
-- There was an error in the style so that boundary=postal_code used to be
-- imported as place=postcode. That's why relations are allowed to pass here.
-- This can go away in a couple of versions.
IF NEW.class = 'place' and NEW.type = 'postcode' and NEW.osm_type != 'R' THEN
NEW.token_info := null;
RETURN NEW;
END IF; END IF;
-- Compute a preliminary centroid. -- Compute a preliminary centroid.
@@ -1046,9 +1055,7 @@ BEGIN
LOOP LOOP
UPDATE placex SET linked_place_id = NEW.place_id WHERE place_id = linked_node_id; UPDATE placex SET linked_place_id = NEW.place_id WHERE place_id = linked_node_id;
{% if 'search_name' in db.tables %} {% if 'search_name' in db.tables %}
IF OLD.indexed_status > 1 THEN
DELETE FROM search_name WHERE place_id = linked_node_id; DELETE FROM search_name WHERE place_id = linked_node_id;
END IF;
{% endif %} {% endif %}
END LOOP; END LOOP;
END IF; END IF;
@@ -1197,6 +1204,11 @@ BEGIN
-- reset the address rank if necessary. -- reset the address rank if necessary.
UPDATE placex set linked_place_id = NEW.place_id, indexed_status = 2 UPDATE placex set linked_place_id = NEW.place_id, indexed_status = 2
WHERE place_id = location.place_id; WHERE place_id = location.place_id;
-- ensure that those places are not found anymore
{% if 'search_name' in db.tables %}
DELETE FROM search_name WHERE place_id = location.place_id;
{% endif %}
PERFORM deleteLocationArea(NEW.partition, location.place_id, NEW.rank_search);
SELECT wikipedia, importance SELECT wikipedia, importance
FROM compute_importance(location.extratags, NEW.country_code, FROM compute_importance(location.extratags, NEW.country_code,
@@ -1207,7 +1219,7 @@ BEGIN
IF linked_importance is not null AND IF linked_importance is not null AND
(NEW.importance is null or NEW.importance < linked_importance) (NEW.importance is null or NEW.importance < linked_importance)
THEN THEN
NEW.importance := linked_importance; NEW.importance = linked_importance;
END IF; END IF;
ELSE ELSE
-- No linked place? As a last resort check if the boundary is tagged with -- No linked place? As a last resort check if the boundary is tagged with
@@ -1249,7 +1261,7 @@ BEGIN
LIMIT 1 LIMIT 1
LOOP LOOP
IF location.osm_id = NEW.osm_id THEN IF location.osm_id = NEW.osm_id THEN
{% if debug %}RAISE WARNING 'Updating names for country ''%'' with: %', NEW.country_code, NEW.name;{% endif %} {% if debug %}RAISE WARNING 'Updating names for country '%' with: %', NEW.country_code, NEW.name;{% endif %}
UPDATE country_name SET derived_name = NEW.name WHERE country_code = NEW.country_code; UPDATE country_name SET derived_name = NEW.name WHERE country_code = NEW.country_code;
END IF; END IF;
END LOOP; END LOOP;
@@ -1274,6 +1286,8 @@ BEGIN
END IF; END IF;
ELSEIF NEW.rank_address > 25 THEN ELSEIF NEW.rank_address > 25 THEN
max_rank := 25; max_rank := 25;
ELSEIF NEW.class in ('place','boundary') and NEW.type in ('postcode','postal_code') THEN
max_rank := NEW.rank_search;
ELSE ELSE
max_rank := NEW.rank_address; max_rank := NEW.rank_address;
END IF; END IF;
@@ -1288,10 +1302,10 @@ BEGIN
NEW.postcode := coalesce(token_get_postcode(NEW.token_info), NEW.postcode); NEW.postcode := coalesce(token_get_postcode(NEW.token_info), NEW.postcode);
-- if we have a name add this to the name search table -- if we have a name add this to the name search table
name_vector := token_get_name_search_tokens(NEW.token_info); IF NEW.name IS NOT NULL THEN
IF array_length(name_vector, 1) is not NULL THEN
-- Initialise the name vector using our name -- Initialise the name vector using our name
NEW.name := add_default_place_name(NEW.country_code, NEW.name); NEW.name := add_default_place_name(NEW.country_code, NEW.name);
name_vector := token_get_name_search_tokens(NEW.token_info);
IF NEW.rank_search <= 25 and NEW.rank_address > 0 THEN IF NEW.rank_search <= 25 and NEW.rank_address > 0 THEN
result := add_location(NEW.place_id, NEW.country_code, NEW.partition, result := add_location(NEW.place_id, NEW.country_code, NEW.partition,
@@ -1346,16 +1360,15 @@ CREATE OR REPLACE FUNCTION placex_delete()
AS $$ AS $$
DECLARE DECLARE
b BOOLEAN; b BOOLEAN;
result INT;
classtable TEXT; classtable TEXT;
BEGIN BEGIN
-- RAISE WARNING 'placex_delete % %',OLD.osm_type,OLD.osm_id; -- RAISE WARNING 'placex_delete % %',OLD.osm_type,OLD.osm_id;
IF OLD.linked_place_id is null THEN IF OLD.linked_place_id is null THEN
UPDATE placex update placex set linked_place_id = null, indexed_status = 2 where linked_place_id = OLD.place_id and indexed_status = 0;
SET linked_place_id = NULL, {% if debug %}RAISE WARNING 'placex_delete:01 % %',OLD.osm_type,OLD.osm_id;{% endif %}
indexed_status = CASE WHEN indexed_status = 0 THEN 2 ELSE indexed_status END update placex set linked_place_id = null where linked_place_id = OLD.place_id;
WHERE linked_place_id = OLD.place_id; {% if debug %}RAISE WARNING 'placex_delete:02 % %',OLD.osm_type,OLD.osm_id;{% endif %}
ELSE ELSE
update placex set indexed_status = 2 where place_id = OLD.linked_place_id and indexed_status = 0; update placex set indexed_status = 2 where place_id = OLD.linked_place_id and indexed_status = 0;
END IF; END IF;
@@ -1379,7 +1392,6 @@ BEGIN
-- reparenting also for OSM Interpolation Lines (and for Tiger?) -- reparenting also for OSM Interpolation Lines (and for Tiger?)
update location_property_osmline set indexed_status = 2 where indexed_status = 0 and parent_place_id = OLD.place_id; update location_property_osmline set indexed_status = 2 where indexed_status = 0 and parent_place_id = OLD.place_id;
UPDATE location_postcodes SET indexed_status = 2 WHERE parent_place_id = OLD.place_id;
END IF; END IF;
{% if debug %}RAISE WARNING 'placex_delete:08 % %',OLD.osm_type,OLD.osm_id;{% endif %} {% if debug %}RAISE WARNING 'placex_delete:08 % %',OLD.osm_type,OLD.osm_id;{% endif %}
@@ -1405,16 +1417,15 @@ BEGIN
-- remove from tables for special search -- remove from tables for special search
classtable := 'place_classtype_' || OLD.class || '_' || OLD.type; classtable := 'place_classtype_' || OLD.class || '_' || OLD.type;
SELECT count(*) INTO result SELECT count(*)>0 FROM pg_tables WHERE tablename = classtable and schemaname = current_schema() INTO b;
FROM pg_tables IF b THEN
WHERE classtable NOT SIMILAR TO '%\W%'
AND tablename = classtable and schemaname = current_schema();
IF result > 0 THEN
EXECUTE 'DELETE FROM ' || classtable::regclass || ' WHERE place_id = $1' USING OLD.place_id; EXECUTE 'DELETE FROM ' || classtable::regclass || ' WHERE place_id = $1' USING OLD.place_id;
END IF; END IF;
{% if debug %}RAISE WARNING 'placex_delete:12 % %',OLD.osm_type,OLD.osm_id;{% endif %} {% if debug %}RAISE WARNING 'placex_delete:12 % %',OLD.osm_type,OLD.osm_id;{% endif %}
UPDATE location_postcode SET indexed_status = 2 WHERE parent_place_id = OLD.place_id;
RETURN OLD; RETURN OLD;
END; END;

View File

@@ -2,10 +2,10 @@
-- --
-- This file is part of Nominatim. (https://nominatim.org) -- This file is part of Nominatim. (https://nominatim.org)
-- --
-- Copyright (C) 2025 by the Nominatim developer community. -- Copyright (C) 2022 by the Nominatim developer community.
-- For a full list of authors see the git log. -- For a full list of authors see the git log.
-- Trigger functions for location_postcodes table. -- Trigger functions for location_postcode table.
-- Trigger for updates of location_postcode -- Trigger for updates of location_postcode
@@ -13,7 +13,7 @@
-- Computes the parent object the postcode most likely refers to. -- Computes the parent object the postcode most likely refers to.
-- This will be the place that determines the address displayed when -- This will be the place that determines the address displayed when
-- searching for this postcode. -- searching for this postcode.
CREATE OR REPLACE FUNCTION postcodes_update() CREATE OR REPLACE FUNCTION postcode_update()
RETURNS TRIGGER RETURNS TRIGGER
AS $$ AS $$
DECLARE DECLARE
@@ -28,10 +28,13 @@ BEGIN
partition := get_partition(NEW.country_code); partition := get_partition(NEW.country_code);
SELECT * FROM get_postcode_rank(NEW.country_code, NEW.postcode)
INTO NEW.rank_search, NEW.rank_address;
NEW.parent_place_id = 0; NEW.parent_place_id = 0;
FOR location IN FOR location IN
SELECT place_id SELECT place_id
FROM getNearFeatures(partition, NEW.centroid, NEW.centroid, NEW.rank_search) FROM getNearFeatures(partition, NEW.geometry, NEW.geometry, NEW.rank_search)
WHERE NOT isguess ORDER BY rank_address DESC, distance asc LIMIT 1 WHERE NOT isguess ORDER BY rank_address DESC, distance asc LIMIT 1
LOOP LOOP
NEW.parent_place_id = location.place_id; NEW.parent_place_id = location.place_id;
@@ -42,89 +45,3 @@ END;
$$ $$
LANGUAGE plpgsql; LANGUAGE plpgsql;
CREATE OR REPLACE FUNCTION postcodes_delete()
RETURNS TRIGGER
AS $$
BEGIN
{% if not disable_diff_updates %}
UPDATE placex p SET indexed_status = 2
WHERE p.postcode = OLD.postcode AND ST_Intersects(OLD.geometry, p.geometry)
AND indexed_status = 0;
{% endif %}
RETURN OLD;
END;
$$
LANGUAGE plpgsql;
CREATE OR REPLACE FUNCTION postcodes_insert()
RETURNS TRIGGER
AS $$
DECLARE
existing RECORD;
BEGIN
IF NEW.osm_id is not NULL THEN
-- postcode area, remove existing from same OSM object
SELECT * INTO existing FROM location_postcodes p
WHERE p.osm_id = NEW.osm_id;
IF existing.place_id is not NULL THEN
IF existing.postcode != NEW.postcode or existing.country_code != NEW.country_code THEN
DELETE FROM location_postcodes p WHERE p.osm_id = NEW.osm_id;
existing := NULL;
END IF;
END IF;
END IF;
IF existing is NULL THEN
SELECT * INTO existing FROM location_postcodes p
WHERE p.country_code = NEW.country_code AND p.postcode = NEW.postcode;
IF existing.postcode is NULL THEN
{% if not disable_diff_updates %}
UPDATE placex p SET indexed_status = 2
WHERE ST_Intersects(NEW.geometry, p.geometry)
AND indexed_status = 0
AND p.rank_address >= 22 AND not p.address ? 'postcode';
{% endif %}
-- new entry, just insert
NEW.indexed_status := 1;
NEW.place_id := nextval('seq_place');
RETURN NEW;
END IF;
END IF;
-- update: only when there are changes
IF coalesce(NEW.osm_id, -1) != coalesce(existing.osm_id, -1)
OR (NEW.osm_id is not null AND NEW.geometry::text != existing.geometry::text)
OR (NEW.osm_id is null
AND (abs(ST_X(existing.centroid) - ST_X(NEW.centroid)) > 0.0000001
OR abs(ST_Y(existing.centroid) - ST_Y(NEW.centroid)) > 0.0000001))
THEN
{% if not disable_diff_updates %}
UPDATE placex p SET indexed_status = 2
WHERE ST_Intersects(ST_Difference(NEW.geometry, existing.geometry), p.geometry)
AND indexed_status = 0
AND p.rank_address >= 22 AND not p.address ? 'postcode';
UPDATE placex p SET indexed_status = 2
WHERE ST_Intersects(ST_Difference(existing.geometry, NEW.geometry), p.geometry)
AND indexed_status = 0
AND p.postcode = OLD.postcode;
{% endif %}
UPDATE location_postcodes p
SET osm_id = NEW.osm_id,
indexed_status = 2,
centroid = NEW.centroid,
geometry = NEW.geometry
WHERE p.country_code = NEW.country_code AND p.postcode = NEW.postcode;
END IF;
RETURN NULL;
END;
$$
LANGUAGE plpgsql;

View File

@@ -2,7 +2,7 @@
-- --
-- This file is part of Nominatim. (https://nominatim.org) -- This file is part of Nominatim. (https://nominatim.org)
-- --
-- Copyright (C) 2025 by the Nominatim developer community. -- Copyright (C) 2022 by the Nominatim developer community.
-- For a full list of authors see the git log. -- For a full list of authors see the git log.
-- Functions related to search and address ranks -- Functions related to search and address ranks
@@ -114,6 +114,66 @@ $$
LANGUAGE plpgsql IMMUTABLE PARALLEL SAFE; LANGUAGE plpgsql IMMUTABLE PARALLEL SAFE;
-- Guess a ranking for postcodes from country and postcode format.
CREATE OR REPLACE FUNCTION get_postcode_rank(country_code VARCHAR(2), postcode TEXT,
OUT rank_search SMALLINT,
OUT rank_address SMALLINT)
AS $$
DECLARE
part TEXT;
BEGIN
rank_search := 30;
rank_address := 30;
postcode := upper(postcode);
IF country_code = 'gb' THEN
IF postcode ~ '^([A-Z][A-Z]?[0-9][0-9A-Z]? [0-9][A-Z][A-Z])$' THEN
rank_search := 25;
rank_address := 5;
ELSEIF postcode ~ '^([A-Z][A-Z]?[0-9][0-9A-Z]? [0-9])$' THEN
rank_search := 23;
rank_address := 5;
ELSEIF postcode ~ '^([A-Z][A-Z]?[0-9][0-9A-Z])$' THEN
rank_search := 21;
rank_address := 5;
END IF;
ELSEIF country_code = 'sg' THEN
IF postcode ~ '^([0-9]{6})$' THEN
rank_search := 25;
rank_address := 11;
END IF;
ELSEIF country_code = 'de' THEN
IF postcode ~ '^([0-9]{5})$' THEN
rank_search := 21;
rank_address := 11;
END IF;
ELSE
-- Guess at the postcode format and coverage (!)
IF postcode ~ '^[A-Z0-9]{1,5}$' THEN -- Probably too short to be very local
rank_search := 21;
rank_address := 11;
ELSE
-- Does it look splitable into and area and local code?
part := substring(postcode from '^([- :A-Z0-9]+)([- :][A-Z0-9]+)$');
IF part IS NOT NULL THEN
rank_search := 25;
rank_address := 11;
ELSEIF postcode ~ '^[- :A-Z0-9]{6,}$' THEN
rank_search := 21;
rank_address := 11;
END IF;
END IF;
END IF;
END;
$$
LANGUAGE plpgsql IMMUTABLE PARALLEL SAFE;
-- Get standard search and address rank for an object. -- Get standard search and address rank for an object.
-- --
-- \param country Two-letter country code where the object is in. -- \param country Two-letter country code where the object is in.
@@ -138,7 +198,12 @@ AS $$
DECLARE DECLARE
classtype TEXT; classtype TEXT;
BEGIN BEGIN
IF extended_type = 'N' AND place_class = 'highway' THEN IF place_class in ('place','boundary')
and place_type in ('postcode','postal_code')
THEN
SELECT * INTO search_rank, address_rank
FROM get_postcode_rank(country, postcode);
ELSEIF extended_type = 'N' AND place_class = 'highway' THEN
search_rank = 30; search_rank = 30;
address_rank = 30; address_rank = 30;
ELSEIF place_class = 'landuse' AND extended_type != 'A' THEN ELSEIF place_class = 'landuse' AND extended_type != 'A' THEN

View File

@@ -2,7 +2,7 @@
-- --
-- This file is part of Nominatim. (https://nominatim.org) -- This file is part of Nominatim. (https://nominatim.org)
-- --
-- Copyright (C) 2026 by the Nominatim developer community. -- Copyright (C) 2025 by the Nominatim developer community.
-- For a full list of authors see the git log. -- For a full list of authors see the git log.
-- Assorted helper functions for the triggers. -- Assorted helper functions for the triggers.
@@ -46,10 +46,10 @@ DECLARE
r INTEGER[]; r INTEGER[];
BEGIN BEGIN
IF array_upper(a, 1) IS NULL THEN IF array_upper(a, 1) IS NULL THEN
RETURN COALESCE(b, '{}'::INTEGER[]); RETURN b;
END IF; END IF;
IF array_upper(b, 1) IS NULL THEN IF array_upper(b, 1) IS NULL THEN
RETURN COALESCE(a, '{}'::INTEGER[]); RETURN a;
END IF; END IF;
r := a; r := a;
FOR i IN 1..array_upper(b, 1) LOOP FOR i IN 1..array_upper(b, 1) LOOP
@@ -139,46 +139,37 @@ $$
LANGUAGE plpgsql IMMUTABLE PARALLEL SAFE; LANGUAGE plpgsql IMMUTABLE PARALLEL SAFE;
-- Find the best-matching postcode for the given geometry -- Find the nearest artificial postcode for the given geometry.
-- TODO For areas there should not be more than two inside the geometry.
CREATE OR REPLACE FUNCTION get_nearest_postcode(country VARCHAR(2), geom GEOMETRY) CREATE OR REPLACE FUNCTION get_nearest_postcode(country VARCHAR(2), geom GEOMETRY)
RETURNS TEXT RETURNS TEXT
AS $$ AS $$
DECLARE DECLARE
outcode TEXT; outcode TEXT;
cnt INTEGER; cnt INTEGER;
location RECORD;
BEGIN BEGIN
-- If the geometry is an area then only one postcode must be within -- If the geometry is an area then only one postcode must be within
-- that area, otherwise consider the area as not having a postcode. -- that area, otherwise consider the area as not having a postcode.
IF ST_GeometryType(geom) in ('ST_Polygon','ST_MultiPolygon') THEN IF ST_GeometryType(geom) in ('ST_Polygon','ST_MultiPolygon') THEN
SELECT min(postcode), count(*) FROM SELECT min(postcode), count(*) FROM
(SELECT postcode FROM location_postcodes (SELECT postcode FROM location_postcode
WHERE geom && location_postcodes.geometry -- want to use the index WHERE ST_Contains(geom, location_postcode.geometry) LIMIT 2) sub
AND ST_Contains(geom, location_postcodes.centroid)
AND country_code = country
LIMIT 2) sub
INTO outcode, cnt; INTO outcode, cnt;
IF cnt = 1 THEN IF cnt = 1 THEN
RETURN outcode; RETURN outcode;
END IF; ELSE
RETURN null; RETURN null;
END IF; END IF;
END IF;
-- Otherwise: be fully within the coverage area of a postcode SELECT postcode FROM location_postcode
FOR location IN WHERE ST_DWithin(geom, location_postcode.geometry, 0.05)
SELECT postcode AND location_postcode.country_code = country
FROM location_postcodes p ORDER BY ST_Distance(geom, location_postcode.geometry) LIMIT 1
WHERE ST_Covers(p.geometry, geom) INTO outcode;
AND p.country_code = country
ORDER BY osm_id is null, ST_Distance(p.centroid, geom)
LIMIT 1
LOOP
RETURN location.postcode;
END LOOP;
RETURN NULL; RETURN outcode;
END; END;
$$ $$
LANGUAGE plpgsql STABLE PARALLEL SAFE; LANGUAGE plpgsql STABLE PARALLEL SAFE;
@@ -323,17 +314,6 @@ END;
$$ $$
LANGUAGE plpgsql; LANGUAGE plpgsql;
-- Return the bounding box of the geometry buffered by the given number
-- of meters.
CREATE OR REPLACE FUNCTION expand_by_meters(geom GEOMETRY, meters FLOAT)
RETURNS GEOMETRY
AS $$
SELECT ST_Envelope(ST_Buffer(geom::geography, meters, 1)::geometry)
$$
LANGUAGE sql IMMUTABLE PARALLEL SAFE;
-- Create a bounding box with an extent computed from the radius (in meters) -- Create a bounding box with an extent computed from the radius (in meters)
-- which in turn is derived from the given search rank. -- which in turn is derived from the given search rank.
CREATE OR REPLACE FUNCTION place_node_fuzzy_area(geom GEOMETRY, rank_search INTEGER) CREATE OR REPLACE FUNCTION place_node_fuzzy_area(geom GEOMETRY, rank_search INTEGER)
@@ -352,7 +332,9 @@ BEGIN
radius := 1000; radius := 1000;
END IF; END IF;
RETURN expand_by_meters(geom, radius); RETURN ST_Envelope(ST_Collect(
ST_Project(geom::geography, radius, 0.785398)::geometry,
ST_Project(geom::geography, radius, 3.9269908)::geometry));
END; END;
$$ $$
LANGUAGE plpgsql IMMUTABLE PARALLEL SAFE; LANGUAGE plpgsql IMMUTABLE PARALLEL SAFE;
@@ -368,6 +350,8 @@ CREATE OR REPLACE FUNCTION add_location(place_id BIGINT, country_code varchar(2)
DECLARE DECLARE
postcode TEXT; postcode TEXT;
BEGIN BEGIN
PERFORM deleteLocationArea(partition, place_id, rank_search);
-- add postcode only if it contains a single entry, i.e. ignore postcode lists -- add postcode only if it contains a single entry, i.e. ignore postcode lists
postcode := NULL; postcode := NULL;
IF in_postcode is not null AND in_postcode not similar to '%(,|;)%' THEN IF in_postcode is not null AND in_postcode not similar to '%(,|;)%' THEN

View File

@@ -1,47 +0,0 @@
-- SPDX-License-Identifier: GPL-2.0-only
--
-- This file is part of Nominatim. (https://nominatim.org)
--
-- Copyright (C) 2026 by the Nominatim developer community.
-- For a full list of authors see the git log.
--
-- Grant read-only access to the web user for all Nominatim tables.
-- Core tables
GRANT SELECT ON import_status TO "{{config.DATABASE_WEBUSER}}";
GRANT SELECT ON country_name TO "{{config.DATABASE_WEBUSER}}";
GRANT SELECT ON nominatim_properties TO "{{config.DATABASE_WEBUSER}}";
-- Location tables
GRANT SELECT ON location_property_tiger TO "{{config.DATABASE_WEBUSER}}";
GRANT SELECT ON location_property_osmline TO "{{config.DATABASE_WEBUSER}}";
GRANT SELECT ON location_postcodes TO "{{config.DATABASE_WEBUSER}}";
-- Search tables
{% if not db.reverse_only %}
GRANT SELECT ON search_name TO "{{config.DATABASE_WEBUSER}}";
{% endif %}
-- Main place tables
GRANT SELECT ON placex TO "{{config.DATABASE_WEBUSER}}";
GRANT SELECT ON place_addressline TO "{{config.DATABASE_WEBUSER}}";
GRANT SELECT ON placex_entrance TO "{{config.DATABASE_WEBUSER}}";
-- Error/delete tracking tables
GRANT SELECT ON import_polygon_error TO "{{config.DATABASE_WEBUSER}}";
GRANT SELECT ON import_polygon_delete TO "{{config.DATABASE_WEBUSER}}";
-- Country grid
GRANT SELECT ON country_osm_grid TO "{{config.DATABASE_WEBUSER}}";
-- Tokenizer tables (word table)
{% if 'word' in db.tables %}
GRANT SELECT ON word TO "{{config.DATABASE_WEBUSER}}";
{% endif %}
-- Special phrase tables
{% for table in db.tables %}
{% if table.startswith('place_classtype_') %}
GRANT SELECT ON {{ table }} TO "{{config.DATABASE_WEBUSER}}";
{% endif %}
{% endfor %}

View File

@@ -2,7 +2,7 @@
-- --
-- This file is part of Nominatim. (https://nominatim.org) -- This file is part of Nominatim. (https://nominatim.org)
-- --
-- Copyright (C) 2025 by the Nominatim developer community. -- Copyright (C) 2022 by the Nominatim developer community.
-- For a full list of authors see the git log. -- For a full list of authors see the git log.
-- Indices used only during search and update. -- Indices used only during search and update.
@@ -21,25 +21,30 @@ CREATE INDEX IF NOT EXISTS idx_placex_parent_place_id
ON placex USING BTREE (parent_place_id) {{db.tablespace.search_index}} ON placex USING BTREE (parent_place_id) {{db.tablespace.search_index}}
WHERE parent_place_id IS NOT NULL; WHERE parent_place_id IS NOT NULL;
--- ---
-- Used to find postcode areas after a search in location_postcode.
CREATE INDEX IF NOT EXISTS idx_placex_postcode_areas
ON placex USING BTREE (country_code, postcode) {{db.tablespace.search_index}}
WHERE osm_type = 'R' AND class = 'boundary' AND type = 'postal_code';
---
CREATE INDEX IF NOT EXISTS idx_placex_geometry ON placex CREATE INDEX IF NOT EXISTS idx_placex_geometry ON placex
USING GIST (geometry) {{db.tablespace.search_index}}; USING GIST (geometry) {{db.tablespace.search_index}};
---
-- Index is needed during import but can be dropped as soon as a full -- Index is needed during import but can be dropped as soon as a full
-- geometry index is in place. The partial index is almost as big as the full -- geometry index is in place. The partial index is almost as big as the full
-- index. -- index.
---
DROP INDEX IF EXISTS idx_placex_geometry_lower_rank_ways; DROP INDEX IF EXISTS idx_placex_geometry_lower_rank_ways;
--- ---
CREATE INDEX IF NOT EXISTS idx_placex_geometry_reverse_lookupPolygon CREATE INDEX IF NOT EXISTS idx_placex_geometry_reverse_lookupPolygon
ON placex USING gist (geometry) {{db.tablespace.search_index}} ON placex USING gist (geometry) {{db.tablespace.search_index}}
WHERE St_GeometryType(geometry) in ('ST_Polygon', 'ST_MultiPolygon') WHERE St_GeometryType(geometry) in ('ST_Polygon', 'ST_MultiPolygon')
AND rank_address between 4 and 25 AND rank_address between 4 and 25 AND type != 'postcode'
AND name is not null AND indexed_status = 0 AND linked_place_id is null; AND name is not null AND indexed_status = 0 AND linked_place_id is null;
--- ---
-- used in reverse large area lookup -- used in reverse large area lookup
CREATE INDEX IF NOT EXISTS idx_placex_geometry_reverse_lookupPlaceNode CREATE INDEX IF NOT EXISTS idx_placex_geometry_reverse_lookupPlaceNode
ON placex USING gist (ST_Buffer(geometry, reverse_place_diameter(rank_search))) ON placex USING gist (ST_Buffer(geometry, reverse_place_diameter(rank_search)))
{{db.tablespace.search_index}} {{db.tablespace.search_index}}
WHERE rank_address between 4 and 25 WHERE rank_address between 4 and 25 AND type != 'postcode'
AND name is not null AND linked_place_id is null AND osm_type = 'N'; AND name is not null AND linked_place_id is null AND osm_type = 'N';
--- ---
CREATE INDEX IF NOT EXISTS idx_osmline_parent_place_id CREATE INDEX IF NOT EXISTS idx_osmline_parent_place_id
@@ -48,6 +53,9 @@ CREATE INDEX IF NOT EXISTS idx_osmline_parent_place_id
--- ---
CREATE INDEX IF NOT EXISTS idx_osmline_parent_osm_id CREATE INDEX IF NOT EXISTS idx_osmline_parent_osm_id
ON location_property_osmline USING BTREE (osm_id) {{db.tablespace.search_index}}; ON location_property_osmline USING BTREE (osm_id) {{db.tablespace.search_index}};
---
CREATE INDEX IF NOT EXISTS idx_postcode_postcode
ON location_postcode USING BTREE (postcode) {{db.tablespace.search_index}};
{% if drop %} {% if drop %}
--- ---
@@ -74,8 +82,8 @@ CREATE INDEX IF NOT EXISTS idx_osmline_parent_osm_id
deferred BOOLEAN deferred BOOLEAN
); );
--- ---
CREATE INDEX IF NOT EXISTS idx_location_postcodes_parent_place_id CREATE INDEX IF NOT EXISTS idx_location_postcode_parent_place_id
ON location_postcodes USING BTREE (parent_place_id) {{db.tablespace.address_index}}; ON location_postcode USING BTREE (parent_place_id) {{db.tablespace.address_index}};
{% endif %} {% endif %}
-- Indices only needed for search. -- Indices only needed for search.

View File

@@ -2,48 +2,36 @@
-- --
-- This file is part of Nominatim. (https://nominatim.org) -- This file is part of Nominatim. (https://nominatim.org)
-- --
-- Copyright (C) 2026 by the Nominatim developer community. -- Copyright (C) 2022 by the Nominatim developer community.
-- For a full list of authors see the git log. -- For a full list of authors see the git log.
drop table IF EXISTS search_name_blank CASCADE; drop table IF EXISTS search_name_blank CASCADE;
CREATE TABLE search_name_blank ( CREATE TABLE search_name_blank (
place_id BIGINT NOT NULL, place_id BIGINT,
address_rank smallint NOT NULL, address_rank smallint,
name_vector integer[] NOT NULL, name_vector integer[],
centroid GEOMETRY(Geometry, 4326) NOT NULL centroid GEOMETRY(Geometry, 4326)
); );
{% for partition in db.partitions %} {% for partition in db.partitions %}
CREATE TABLE location_area_large_{{ partition }} () INHERITS (location_area_large) {{db.tablespace.address_data}}; CREATE TABLE location_area_large_{{ partition }} () INHERITS (location_area_large) {{db.tablespace.address_data}};
CREATE INDEX idx_location_area_large_{{ partition }}_place_id CREATE INDEX idx_location_area_large_{{ partition }}_place_id ON location_area_large_{{ partition }} USING BTREE (place_id) {{db.tablespace.address_index}};
ON location_area_large_{{ partition }} CREATE INDEX idx_location_area_large_{{ partition }}_geometry ON location_area_large_{{ partition }} USING GIST (geometry) {{db.tablespace.address_index}};
USING BTREE (place_id) {{db.tablespace.address_index}};
CREATE INDEX idx_location_area_large_{{ partition }}_geometry
ON location_area_large_{{ partition }}
USING GIST (geometry) {{db.tablespace.address_index}};
CREATE TABLE search_name_{{ partition }} () INHERITS (search_name_blank) {{db.tablespace.address_data}}; CREATE TABLE search_name_{{ partition }} () INHERITS (search_name_blank) {{db.tablespace.address_data}};
CREATE UNIQUE INDEX idx_search_name_{{ partition }}_place_id CREATE INDEX idx_search_name_{{ partition }}_place_id ON search_name_{{ partition }} USING BTREE (place_id) {{db.tablespace.address_index}};
ON search_name_{{ partition }} CREATE INDEX idx_search_name_{{ partition }}_centroid_street ON search_name_{{ partition }} USING GIST (centroid) {{db.tablespace.address_index}} where address_rank between 26 and 27;
USING BTREE (place_id) {{db.tablespace.address_index}}; CREATE INDEX idx_search_name_{{ partition }}_centroid_place ON search_name_{{ partition }} USING GIST (centroid) {{db.tablespace.address_index}} where address_rank between 2 and 25;
CREATE INDEX idx_search_name_{{ partition }}_centroid_street
ON search_name_{{ partition }} USING GIST (centroid) {{db.tablespace.address_index}}
WHERE address_rank between 26 and 27;
CREATE INDEX idx_search_name_{{ partition }}_centroid_place
ON search_name_{{ partition }} USING GIST (centroid) {{db.tablespace.address_index}}
WHERE address_rank between 2 and 25;
DROP TABLE IF EXISTS location_road_{{ partition }}; DROP TABLE IF EXISTS location_road_{{ partition }};
CREATE TABLE location_road_{{ partition }} ( CREATE TABLE location_road_{{ partition }} (
place_id BIGINT NOT NULL, place_id BIGINT,
partition SMALLINT NOT NULL, partition SMALLINT,
country_code VARCHAR(2), country_code VARCHAR(2),
geometry GEOMETRY(Geometry, 4326) NOT NULL geometry GEOMETRY(Geometry, 4326)
) {{db.tablespace.address_data}}; ) {{db.tablespace.address_data}};
CREATE INDEX idx_location_road_{{ partition }}_geometry CREATE INDEX idx_location_road_{{ partition }}_geometry ON location_road_{{ partition }} USING GIST (geometry) {{db.tablespace.address_index}};
ON location_road_{{ partition }} CREATE INDEX idx_location_road_{{ partition }}_place_id ON location_road_{{ partition }} USING BTREE (place_id) {{db.tablespace.address_index}};
USING GIST (geometry) {{db.tablespace.address_index}};
CREATE UNIQUE INDEX idx_location_road_{{ partition }}_place_id
ON location_road_{{ partition }}
USING BTREE (place_id) {{db.tablespace.address_index}};
{% endfor %} {% endfor %}

View File

@@ -2,7 +2,7 @@
-- --
-- This file is part of Nominatim. (https://nominatim.org) -- This file is part of Nominatim. (https://nominatim.org)
-- --
-- Copyright (C) 2025 by the Nominatim developer community. -- Copyright (C) 2022 by the Nominatim developer community.
-- For a full list of authors see the git log. -- For a full list of authors see the git log.
-- insert creates the location tables, creates location indexes if indexed == true -- insert creates the location tables, creates location indexes if indexed == true
@@ -25,9 +25,5 @@ CREATE TRIGGER place_before_delete BEFORE DELETE ON place
CREATE TRIGGER place_before_insert BEFORE INSERT ON place CREATE TRIGGER place_before_insert BEFORE INSERT ON place
FOR EACH ROW EXECUTE PROCEDURE place_insert(); FOR EACH ROW EXECUTE PROCEDURE place_insert();
CREATE TRIGGER location_postcode_before_update BEFORE UPDATE ON location_postcodes CREATE TRIGGER location_postcode_before_update BEFORE UPDATE ON location_postcode
FOR EACH ROW EXECUTE PROCEDURE postcodes_update(); FOR EACH ROW EXECUTE PROCEDURE postcode_update();
CREATE TRIGGER location_postcodes_before_delete BEFORE DELETE ON location_postcodes
FOR EACH ROW EXECUTE PROCEDURE postcodes_delete();
CREATE TRIGGER location_postcodes_before_insert BEFORE INSERT ON location_postcodes
FOR EACH ROW EXECUTE PROCEDURE postcodes_insert();

View File

@@ -2,24 +2,305 @@
-- --
-- This file is part of Nominatim. (https://nominatim.org) -- This file is part of Nominatim. (https://nominatim.org)
-- --
-- Copyright (C) 2026 by the Nominatim developer community. -- Copyright (C) 2022 by the Nominatim developer community.
-- For a full list of authors see the git log. -- For a full list of authors see the git log.
drop table if exists import_status;
CREATE TABLE import_status (
lastimportdate timestamp with time zone NOT NULL,
sequence_id integer,
indexed boolean
);
GRANT SELECT ON import_status TO "{{config.DATABASE_WEBUSER}}" ;
drop table if exists import_osmosis_log;
CREATE TABLE import_osmosis_log (
batchend timestamp,
batchseq integer,
batchsize bigint,
starttime timestamp,
endtime timestamp,
event text
);
CREATE TABLE new_query_log (
type text,
starttime timestamp,
ipaddress text,
useragent text,
language text,
query text,
searchterm text,
endtime timestamp,
results integer,
format text,
secret text
);
CREATE INDEX idx_new_query_log_starttime ON new_query_log USING BTREE (starttime);
GRANT INSERT ON new_query_log TO "{{config.DATABASE_WEBUSER}}" ;
GRANT UPDATE ON new_query_log TO "{{config.DATABASE_WEBUSER}}" ;
GRANT SELECT ON new_query_log TO "{{config.DATABASE_WEBUSER}}" ;
GRANT SELECT ON TABLE country_name TO "{{config.DATABASE_WEBUSER}}";
DROP TABLE IF EXISTS nominatim_properties;
CREATE TABLE nominatim_properties (
property TEXT NOT NULL,
value TEXT
);
GRANT SELECT ON TABLE nominatim_properties TO "{{config.DATABASE_WEBUSER}}";
drop table IF EXISTS location_area CASCADE;
CREATE TABLE location_area (
place_id BIGINT,
keywords INTEGER[],
partition SMALLINT,
rank_search SMALLINT NOT NULL,
rank_address SMALLINT NOT NULL,
country_code VARCHAR(2),
isguess BOOL,
postcode TEXT,
centroid GEOMETRY(Point, 4326),
geometry GEOMETRY(Geometry, 4326)
);
CREATE TABLE location_area_large () INHERITS (location_area);
DROP TABLE IF EXISTS location_area_country;
CREATE TABLE location_area_country (
place_id BIGINT,
country_code varchar(2),
geometry GEOMETRY(Geometry, 4326)
) {{db.tablespace.address_data}};
CREATE INDEX idx_location_area_country_geometry ON location_area_country USING GIST (geometry) {{db.tablespace.address_index}};
CREATE TABLE location_property_tiger (
place_id BIGINT,
parent_place_id BIGINT,
startnumber INTEGER,
endnumber INTEGER,
step SMALLINT,
partition SMALLINT,
linegeo GEOMETRY,
postcode TEXT);
GRANT SELECT ON location_property_tiger TO "{{config.DATABASE_WEBUSER}}";
drop table if exists location_property_osmline;
CREATE TABLE location_property_osmline (
place_id BIGINT NOT NULL,
osm_id BIGINT,
parent_place_id BIGINT,
geometry_sector INTEGER,
indexed_date TIMESTAMP,
startnumber INTEGER,
endnumber INTEGER,
step SMALLINT,
partition SMALLINT,
indexed_status SMALLINT,
linegeo GEOMETRY,
address HSTORE,
token_info JSONB, -- custom column for tokenizer use only
postcode TEXT,
country_code VARCHAR(2)
){{db.tablespace.search_data}};
CREATE UNIQUE INDEX idx_osmline_place_id ON location_property_osmline USING BTREE (place_id) {{db.tablespace.search_index}};
CREATE INDEX idx_osmline_geometry_sector ON location_property_osmline USING BTREE (geometry_sector) {{db.tablespace.address_index}};
CREATE INDEX idx_osmline_linegeo ON location_property_osmline USING GIST (linegeo) {{db.tablespace.search_index}}
WHERE startnumber is not null;
GRANT SELECT ON location_property_osmline TO "{{config.DATABASE_WEBUSER}}";
drop table IF EXISTS search_name;
{% if not db.reverse_only %}
CREATE TABLE search_name (
place_id BIGINT,
importance FLOAT,
search_rank SMALLINT,
address_rank SMALLINT,
name_vector integer[],
nameaddress_vector integer[],
country_code varchar(2),
centroid GEOMETRY(Geometry, 4326)
) {{db.tablespace.search_data}};
CREATE INDEX idx_search_name_place_id ON search_name USING BTREE (place_id) {{db.tablespace.search_index}};
GRANT SELECT ON search_name to "{{config.DATABASE_WEBUSER}}" ;
{% endif %}
drop table IF EXISTS place_addressline;
CREATE TABLE place_addressline (
place_id BIGINT,
address_place_id BIGINT,
distance FLOAT,
cached_rank_address SMALLINT,
fromarea boolean,
isaddress boolean
) {{db.tablespace.search_data}};
CREATE INDEX idx_place_addressline_place_id on place_addressline USING BTREE (place_id) {{db.tablespace.search_index}};
--------- PLACEX - storage for all indexed places -----------------
DROP TABLE IF EXISTS placex;
CREATE TABLE placex (
place_id BIGINT NOT NULL,
parent_place_id BIGINT,
linked_place_id BIGINT,
importance FLOAT,
indexed_date TIMESTAMP,
geometry_sector INTEGER,
rank_address SMALLINT,
rank_search SMALLINT,
partition SMALLINT,
indexed_status SMALLINT,
LIKE place INCLUDING CONSTRAINTS,
wikipedia TEXT, -- calculated wikipedia article name (language:title)
token_info JSONB, -- custom column for tokenizer use only
country_code varchar(2),
housenumber TEXT,
postcode TEXT,
centroid GEOMETRY(Geometry, 4326)
) {{db.tablespace.search_data}};
CREATE UNIQUE INDEX idx_place_id ON placex USING BTREE (place_id) {{db.tablespace.search_index}};
{% for osm_type in ('N', 'W', 'R') %}
CREATE INDEX idx_placex_osmid_{{osm_type | lower}} ON placex
USING BTREE (osm_id) {{db.tablespace.search_index}}
WHERE osm_type = '{{osm_type}}';
{% endfor %}
-- Usage: - removing linkage status on update
-- - lookup linked places for /details
CREATE INDEX idx_placex_linked_place_id ON placex
USING BTREE (linked_place_id) {{db.tablespace.address_index}}
WHERE linked_place_id IS NOT NULL;
-- Usage: - check that admin boundaries do not overtake each other rank-wise
-- - check that place node in a admin boundary with the same address level
-- - boundary is not completely contained in a place area
-- - parenting of large-area or unparentable features
CREATE INDEX idx_placex_geometry_address_area_candidates ON placex
USING gist (geometry) {{db.tablespace.address_index}}
WHERE rank_address between 1 and 25
and ST_GeometryType(geometry) in ('ST_Polygon','ST_MultiPolygon');
-- Usage: - POI is within building with housenumber
CREATE INDEX idx_placex_geometry_buildings ON placex
USING SPGIST (geometry) {{db.tablespace.address_index}}
WHERE address is not null and rank_search = 30
and ST_GeometryType(geometry) in ('ST_Polygon','ST_MultiPolygon');
-- Usage: - linking of similar named places to boundaries
-- - linking of place nodes with same type to boundaries
CREATE INDEX idx_placex_geometry_placenode ON placex
USING SPGIST (geometry) {{db.tablespace.address_index}}
WHERE osm_type = 'N' and rank_search < 26
and class = 'place' and type != 'postcode';
-- Usage: - is node part of a way?
-- - find parent of interpolation spatially
CREATE INDEX idx_placex_geometry_lower_rank_ways ON placex
USING SPGIST (geometry) {{db.tablespace.address_index}}
WHERE osm_type = 'W' and rank_search >= 26;
-- Usage: - linking place nodes by wikidata tag to boundaries
CREATE INDEX idx_placex_wikidata on placex
USING BTREE ((extratags -> 'wikidata')) {{db.tablespace.address_index}}
WHERE extratags ? 'wikidata' and class = 'place'
and osm_type = 'N' and rank_search < 26;
-- The following two indexes function as a todo list for indexing.
CREATE INDEX idx_placex_rank_address_sector ON placex
USING BTREE (rank_address, geometry_sector) {{db.tablespace.address_index}}
WHERE indexed_status > 0;
CREATE INDEX idx_placex_rank_boundaries_sector ON placex
USING BTREE (rank_search, geometry_sector) {{db.tablespace.address_index}}
WHERE class = 'boundary' and type = 'administrative'
and indexed_status > 0;
DROP SEQUENCE IF EXISTS seq_place; DROP SEQUENCE IF EXISTS seq_place;
CREATE SEQUENCE seq_place start 1; CREATE SEQUENCE seq_place start 1;
GRANT SELECT on placex to "{{config.DATABASE_WEBUSER}}" ;
GRANT SELECT on place_addressline to "{{config.DATABASE_WEBUSER}}" ;
GRANT SELECT ON planet_osm_ways to "{{config.DATABASE_WEBUSER}}" ;
GRANT SELECT ON planet_osm_rels to "{{config.DATABASE_WEBUSER}}" ;
GRANT SELECT on location_area to "{{config.DATABASE_WEBUSER}}" ;
{% include('tables/status.sql') %} -- Table for synthetic postcodes.
{% include('tables/nominatim_properties.sql') %} DROP TABLE IF EXISTS location_postcode;
{% include('tables/location_area.sql') %} CREATE TABLE location_postcode (
{% include('tables/tiger.sql') %} place_id BIGINT,
{% include('tables/interpolation.sql') %} parent_place_id BIGINT,
{% include('tables/search_name.sql') %} rank_search SMALLINT,
{% include('tables/addressline.sql') %} rank_address SMALLINT,
{% include('tables/placex.sql') %} indexed_status SMALLINT,
{% include('tables/postcodes.sql') %} indexed_date TIMESTAMP,
{% include('tables/entrance.sql') %} country_code varchar(2),
{% include('tables/import_reports.sql') %} postcode TEXT,
{% include('tables/importance_tables.sql') %} geometry GEOMETRY(Geometry, 4326)
);
CREATE UNIQUE INDEX idx_postcode_id ON location_postcode USING BTREE (place_id) {{db.tablespace.search_index}};
CREATE INDEX idx_postcode_geometry ON location_postcode USING GIST (geometry) {{db.tablespace.address_index}};
GRANT SELECT ON location_postcode TO "{{config.DATABASE_WEBUSER}}" ;
-- Table to store location of entrance nodes
DROP TABLE IF EXISTS placex_entrance;
CREATE TABLE placex_entrance (
place_id BIGINT NOT NULL,
osm_id BIGINT NOT NULL,
type TEXT NOT NULL,
location GEOMETRY(Point, 4326) NOT NULL,
extratags HSTORE
);
CREATE UNIQUE INDEX idx_placex_entrance_place_id_osm_id ON placex_entrance
USING BTREE (place_id, osm_id) {{db.tablespace.search_index}};
GRANT SELECT ON placex_entrance TO "{{config.DATABASE_WEBUSER}}" ;
-- Create an index on the place table for lookups to populate the entrance
-- table
CREATE INDEX IF NOT EXISTS idx_placex_entrance_lookup ON place
USING BTREE (osm_id)
WHERE class IN ('routing:entrance', 'entrance');
DROP TABLE IF EXISTS import_polygon_error;
CREATE TABLE import_polygon_error (
osm_id BIGINT,
osm_type CHAR(1),
class TEXT NOT NULL,
type TEXT NOT NULL,
name HSTORE,
country_code varchar(2),
updated timestamp,
errormessage text,
prevgeometry GEOMETRY(Geometry, 4326),
newgeometry GEOMETRY(Geometry, 4326)
);
CREATE INDEX idx_import_polygon_error_osmid ON import_polygon_error USING BTREE (osm_type, osm_id);
GRANT SELECT ON import_polygon_error TO "{{config.DATABASE_WEBUSER}}";
DROP TABLE IF EXISTS import_polygon_delete;
CREATE TABLE import_polygon_delete (
osm_id BIGINT,
osm_type CHAR(1),
class TEXT NOT NULL,
type TEXT NOT NULL
);
CREATE INDEX idx_import_polygon_delete_osmid ON import_polygon_delete USING BTREE (osm_type, osm_id);
GRANT SELECT ON import_polygon_delete TO "{{config.DATABASE_WEBUSER}}";
DROP SEQUENCE IF EXISTS file;
CREATE SEQUENCE file start 1;
{% if 'wikimedia_importance' not in db.tables and 'wikipedia_article' not in db.tables %}
-- create dummy tables here, if nothing was imported
CREATE TABLE wikimedia_importance (
language TEXT NOT NULL,
title TEXT NOT NULL,
importance double precision NOT NULL,
wikidata TEXT
) {{db.tablespace.address_data}};
{% endif %}
-- osm2pgsql does not create indexes on the middle tables for Nominatim -- osm2pgsql does not create indexes on the middle tables for Nominatim
-- Add one for lookup of associated street relations. -- Add one for lookup of associated street relations.
@@ -37,3 +318,5 @@ CREATE INDEX planet_osm_rels_relation_members_idx ON planet_osm_rels USING gin(p
CREATE INDEX IF NOT EXISTS idx_place_interpolations CREATE INDEX IF NOT EXISTS idx_place_interpolations
ON place USING gist(geometry) {{db.tablespace.address_index}} ON place USING gist(geometry) {{db.tablespace.address_index}}
WHERE osm_type = 'W' and address ? 'interpolation'; WHERE osm_type = 'W' and address ? 'interpolation';
GRANT SELECT ON table country_osm_grid to "{{config.DATABASE_WEBUSER}}";

View File

@@ -1,20 +0,0 @@
-- SPDX-License-Identifier: GPL-2.0-only
--
-- This file is part of Nominatim. (https://nominatim.org)
--
-- Copyright (C) 2026 by the Nominatim developer community.
-- For a full list of authors see the git log.
DROP TABLE IF EXISTS place_addressline;
CREATE TABLE place_addressline (
place_id BIGINT NOT NULL,
address_place_id BIGINT NOT NULL,
distance FLOAT NOT NULL,
cached_rank_address SMALLINT NOT NULL,
fromarea boolean NOT NULL,
isaddress boolean NOT NULL
) {{db.tablespace.search_data}};
CREATE INDEX idx_place_addressline_place_id ON place_addressline
USING BTREE (place_id) {{db.tablespace.search_index}};

View File

@@ -1,20 +0,0 @@
-- SPDX-License-Identifier: GPL-2.0-only
--
-- This file is part of Nominatim. (https://nominatim.org)
--
-- Copyright (C) 2026 by the Nominatim developer community.
-- For a full list of authors see the git log.
-- Table to store location of entrance nodes
DROP TABLE IF EXISTS placex_entrance;
CREATE TABLE placex_entrance (
place_id BIGINT NOT NULL,
osm_id BIGINT NOT NULL,
type TEXT NOT NULL,
location GEOMETRY(Point, 4326) NOT NULL,
extratags HSTORE
);
CREATE UNIQUE INDEX idx_placex_entrance_place_id_osm_id ON placex_entrance
USING BTREE (place_id, osm_id) {{db.tablespace.search_index}};

View File

@@ -1,35 +0,0 @@
-- SPDX-License-Identifier: GPL-2.0-only
--
-- This file is part of Nominatim. (https://nominatim.org)
--
-- Copyright (C) 2026 by the Nominatim developer community.
-- For a full list of authors see the git log.
DROP TABLE IF EXISTS import_polygon_error;
CREATE TABLE import_polygon_error (
osm_id BIGINT,
osm_type CHAR(1),
class TEXT NOT NULL,
type TEXT NOT NULL,
name HSTORE,
country_code varchar(2),
updated timestamp,
errormessage text,
prevgeometry GEOMETRY(Geometry, 4326),
newgeometry GEOMETRY(Geometry, 4326)
);
CREATE INDEX idx_import_polygon_error_osmid ON import_polygon_error
USING BTREE (osm_type, osm_id);
DROP TABLE IF EXISTS import_polygon_delete;
CREATE TABLE import_polygon_delete (
osm_id BIGINT,
osm_type CHAR(1),
class TEXT NOT NULL,
type TEXT NOT NULL
);
CREATE INDEX idx_import_polygon_delete_osmid ON import_polygon_delete
USING BTREE (osm_type, osm_id);

View File

@@ -1,16 +0,0 @@
-- SPDX-License-Identifier: GPL-2.0-only
--
-- This file is part of Nominatim. (https://nominatim.org)
--
-- Copyright (C) 2026 by the Nominatim developer community.
-- For a full list of authors see the git log.
{% if 'wikimedia_importance' not in db.tables and 'wikipedia_article' not in db.tables %}
-- create dummy tables here if nothing was imported
CREATE TABLE wikimedia_importance (
language TEXT NOT NULL,
title TEXT NOT NULL,
importance double precision NOT NULL,
wikidata TEXT
) {{db.tablespace.address_data}};
{% endif %}

View File

@@ -1,34 +0,0 @@
-- SPDX-License-Identifier: GPL-2.0-only
--
-- This file is part of Nominatim. (https://nominatim.org)
--
-- Copyright (C) 2026 by the Nominatim developer community.
-- For a full list of authors see the git log.
DROP TABLE IF EXISTS location_property_osmline;
CREATE TABLE location_property_osmline (
place_id BIGINT NOT NULL,
osm_id BIGINT NOT NULL,
parent_place_id BIGINT,
geometry_sector INTEGER NOT NULL,
indexed_date TIMESTAMP,
startnumber INTEGER,
endnumber INTEGER,
step SMALLINT,
partition SMALLINT NOT NULL,
indexed_status SMALLINT NOT NULL,
linegeo GEOMETRY NOT NULL,
address HSTORE,
token_info JSONB, -- custom column for tokenizer use only
postcode TEXT,
country_code VARCHAR(2)
){{db.tablespace.search_data}};
CREATE UNIQUE INDEX idx_osmline_place_id ON location_property_osmline
USING BTREE (place_id) {{db.tablespace.search_index}};
CREATE INDEX idx_osmline_geometry_sector ON location_property_osmline
USING BTREE (geometry_sector) {{db.tablespace.address_index}};
CREATE INDEX idx_osmline_linegeo ON location_property_osmline
USING GIST (linegeo) {{db.tablespace.search_index}}
WHERE startnumber is not null;

View File

@@ -1,32 +0,0 @@
-- SPDX-License-Identifier: GPL-2.0-only
--
-- This file is part of Nominatim. (https://nominatim.org)
--
-- Copyright (C) 2026 by the Nominatim developer community.
-- For a full list of authors see the git log.
DROP TABLE IF EXISTS location_area CASCADE;
CREATE TABLE location_area (
place_id BIGINT NOT NULL,
keywords INTEGER[] NOT NULL,
partition SMALLINT NOT NULL,
rank_search SMALLINT NOT NULL,
rank_address SMALLINT NOT NULL,
country_code VARCHAR(2),
isguess BOOL NOT NULL,
postcode TEXT,
centroid GEOMETRY(Point, 4326) NOT NULL,
geometry GEOMETRY(Geometry, 4326) NOT NULL
);
CREATE TABLE location_area_large () INHERITS (location_area);
DROP TABLE IF EXISTS location_area_country;
CREATE TABLE location_area_country (
place_id BIGINT NOT NULL,
country_code varchar(2) NOT NULL,
geometry GEOMETRY(Geometry, 4326) NOT NULL
) {{db.tablespace.address_data}};
CREATE INDEX idx_location_area_country_geometry ON location_area_country
USING GIST (geometry) {{db.tablespace.address_index}};

View File

@@ -1,12 +0,0 @@
-- SPDX-License-Identifier: GPL-2.0-only
--
-- This file is part of Nominatim. (https://nominatim.org)
--
-- Copyright (C) 2026 by the Nominatim developer community.
-- For a full list of authors see the git log.
DROP TABLE IF EXISTS nominatim_properties;
CREATE TABLE nominatim_properties (
property TEXT NOT NULL,
value TEXT
);

View File

@@ -1,87 +0,0 @@
-- SPDX-License-Identifier: GPL-2.0-only
--
-- This file is part of Nominatim. (https://nominatim.org)
--
-- Copyright (C) 2026 by the Nominatim developer community.
-- For a full list of authors see the git log.
-- placex - main table for searchable places
DROP TABLE IF EXISTS placex;
CREATE TABLE placex (
place_id BIGINT NOT NULL,
parent_place_id BIGINT,
linked_place_id BIGINT,
importance FLOAT,
indexed_date TIMESTAMP,
geometry_sector INTEGER NOT NULL,
rank_address SMALLINT NOT NULL,
rank_search SMALLINT NOT NULL,
partition SMALLINT NOT NULL,
indexed_status SMALLINT NOT NULL,
LIKE place INCLUDING CONSTRAINTS,
wikipedia TEXT, -- calculated wikipedia article name (language:title)
token_info JSONB, -- custom column for tokenizer use only
country_code varchar(2),
housenumber TEXT,
postcode TEXT,
centroid GEOMETRY(Geometry, 4326) NOT NULL
) {{db.tablespace.search_data}};
CREATE UNIQUE INDEX idx_place_id ON placex USING BTREE (place_id) {{db.tablespace.search_index}};
{% for osm_type in ('N', 'W', 'R') %}
CREATE INDEX idx_placex_osmid_{{osm_type | lower}} ON placex
USING BTREE (osm_id) {{db.tablespace.search_index}}
WHERE osm_type = '{{osm_type}}';
{% endfor %}
-- Usage: - removing linkage status on update
-- - lookup linked places for /details
CREATE INDEX idx_placex_linked_place_id ON placex
USING BTREE (linked_place_id) {{db.tablespace.address_index}}
WHERE linked_place_id IS NOT NULL;
-- Usage: - check that admin boundaries do not overtake each other rank-wise
-- - check that place node in a admin boundary with the same address level
-- - boundary is not completely contained in a place area
-- - parenting of large-area or unparentable features
CREATE INDEX idx_placex_geometry_address_area_candidates ON placex
USING gist (geometry) {{db.tablespace.address_index}}
WHERE rank_address between 1 and 25
and ST_GeometryType(geometry) in ('ST_Polygon','ST_MultiPolygon');
-- Usage: - POI is within building with housenumber
CREATE INDEX idx_placex_geometry_buildings ON placex
USING SPGIST (geometry) {{db.tablespace.address_index}}
WHERE address is not null and rank_search = 30
and ST_GeometryType(geometry) in ('ST_Polygon','ST_MultiPolygon');
-- Usage: - linking of similar named places to boundaries
-- - linking of place nodes with same type to boundaries
CREATE INDEX idx_placex_geometry_placenode ON placex
USING SPGIST (geometry) {{db.tablespace.address_index}}
WHERE osm_type = 'N' and rank_search < 26 and class = 'place';
-- Usage: - is node part of a way?
-- - find parent of interpolation spatially
CREATE INDEX idx_placex_geometry_lower_rank_ways ON placex
USING SPGIST (geometry) {{db.tablespace.address_index}}
WHERE osm_type = 'W' and rank_search >= 26;
-- Usage: - linking place nodes by wikidata tag to boundaries
CREATE INDEX idx_placex_wikidata on placex
USING BTREE ((extratags -> 'wikidata')) {{db.tablespace.address_index}}
WHERE extratags ? 'wikidata' and class = 'place'
and osm_type = 'N' and rank_search < 26;
-- The following two indexes function as a todo list for indexing.
CREATE INDEX idx_placex_rank_address_sector ON placex
USING BTREE (rank_address, geometry_sector) {{db.tablespace.address_index}}
WHERE indexed_status > 0;
CREATE INDEX idx_placex_rank_boundaries_sector ON placex
USING BTREE (rank_search, geometry_sector) {{db.tablespace.address_index}}
WHERE class = 'boundary' and type = 'administrative'
and indexed_status > 0;

View File

@@ -1,30 +0,0 @@
-- SPDX-License-Identifier: GPL-2.0-only
--
-- This file is part of Nominatim. (https://nominatim.org)
--
-- Copyright (C) 2026 by the Nominatim developer community.
-- For a full list of authors see the git log.
DROP TABLE IF EXISTS location_postcodes;
CREATE TABLE location_postcodes (
place_id BIGINT NOT NULL,
parent_place_id BIGINT,
osm_id BIGINT,
rank_search SMALLINT NOT NULL,
indexed_status SMALLINT NOT NULL,
indexed_date TIMESTAMP,
country_code varchar(2) NOT NULL,
postcode TEXT NOT NULL,
centroid GEOMETRY(Geometry, 4326) NOT NULL,
geometry GEOMETRY(Geometry, 4326) NOT NULL
);
CREATE UNIQUE INDEX idx_location_postcodes_id ON location_postcodes
USING BTREE (place_id) {{db.tablespace.search_index}};
CREATE INDEX idx_location_postcodes_geometry ON location_postcodes
USING GIST (geometry) {{db.tablespace.search_index}};
CREATE INDEX IF NOT EXISTS idx_location_postcodes_postcode ON location_postcodes
USING BTREE (postcode, country_code) {{db.tablespace.search_index}};
CREATE INDEX IF NOT EXISTS idx_location_postcodes_osmid ON location_postcodes
USING BTREE (osm_id) {{db.tablespace.search_index}};

View File

@@ -1,26 +0,0 @@
-- SPDX-License-Identifier: GPL-2.0-only
--
-- This file is part of Nominatim. (https://nominatim.org)
--
-- Copyright (C) 2026 by the Nominatim developer community.
-- For a full list of authors see the git log.
DROP TABLE IF EXISTS search_name;
{% if not create_reverse_only %}
CREATE TABLE search_name (
place_id BIGINT NOT NULL,
importance FLOAT NOT NULL,
search_rank SMALLINT NOT NULL,
address_rank SMALLINT NOT NULL,
name_vector integer[] NOT NULL,
nameaddress_vector integer[] NOT NULL,
country_code varchar(2),
centroid GEOMETRY(Geometry, 4326) NOT NULL
) {{db.tablespace.search_data}};
CREATE UNIQUE INDEX idx_search_name_place_id
ON search_name USING BTREE (place_id) {{db.tablespace.search_index}};
{% endif %}

View File

@@ -1,23 +0,0 @@
-- SPDX-License-Identifier: GPL-2.0-only
--
-- This file is part of Nominatim. (https://nominatim.org)
--
-- Copyright (C) 2026 by the Nominatim developer community.
-- For a full list of authors see the git log.
DROP TABLE IF EXISTS import_status;
CREATE TABLE import_status (
lastimportdate TIMESTAMP WITH TIME ZONE NOT NULL,
sequence_id INTEGER,
indexed BOOLEAN
);
DROP TABLE IF EXISTS import_osmosis_log;
CREATE TABLE import_osmosis_log (
batchend TIMESTAMP,
batchseq INTEGER,
batchsize BIGINT,
starttime TIMESTAMP,
endtime TIMESTAMP,
event TEXT
);

View File

@@ -1,17 +0,0 @@
-- SPDX-License-Identifier: GPL-2.0-only
--
-- This file is part of Nominatim. (https://nominatim.org)
--
-- Copyright (C) 2026 by the Nominatim developer community.
-- For a full list of authors see the git log.
DROP TABLE IF EXISTS location_property_tiger;
CREATE TABLE location_property_tiger (
place_id BIGINT NOT NULL,
parent_place_id BIGINT,
startnumber INTEGER NOT NULL,
endnumber INTEGER NOT NULL,
step SMALLINT NOT NULL,
partition SMALLINT NOT NULL,
linegeo GEOMETRY NOT NULL,
postcode TEXT);

View File

@@ -15,99 +15,6 @@ CREATE TABLE location_property_tiger_import (
step SMALLINT, step SMALLINT,
postcode TEXT); postcode TEXT);
-- Lookup functions for tiger import when update
-- tables are dropped (see gh-issue #2463)
CREATE OR REPLACE FUNCTION getNearestNamedRoadPlaceIdSlow(in_centroid GEOMETRY,
in_token_info JSONB)
RETURNS BIGINT
AS $$
DECLARE
out_place_id BIGINT;
BEGIN
SELECT place_id INTO out_place_id
FROM search_name
WHERE
-- finds rows where name_vector shares elements with search tokens.
token_matches_street(in_token_info, name_vector)
-- limits search area
AND centroid && ST_Expand(in_centroid, 0.015)
AND address_rank BETWEEN 26 AND 27
ORDER BY ST_Distance(centroid, in_centroid) ASC
LIMIT 1;
RETURN out_place_id;
END
$$
LANGUAGE plpgsql;
CREATE OR REPLACE FUNCTION getNearestParallelRoadFeatureSlow(line GEOMETRY)
RETURNS BIGINT
AS $$
DECLARE
r RECORD;
search_diameter FLOAT;
p1 GEOMETRY;
p2 GEOMETRY;
p3 GEOMETRY;
BEGIN
IF ST_GeometryType(line) not in ('ST_LineString') THEN
RETURN NULL;
END IF;
p1 := ST_LineInterpolatePoint(line,0);
p2 := ST_LineInterpolatePoint(line,0.5);
p3 := ST_LineInterpolatePoint(line,1);
search_diameter := 0.0005;
WHILE search_diameter < 0.01 LOOP
FOR r IN
SELECT place_id FROM placex
WHERE ST_DWithin(line, geometry, search_diameter)
AND rank_address BETWEEN 26 AND 27
ORDER BY (ST_distance(geometry, p1)+
ST_distance(geometry, p2)+
ST_distance(geometry, p3)) ASC limit 1
LOOP
RETURN r.place_id;
END LOOP;
search_diameter := search_diameter * 2;
END LOOP;
RETURN NULL;
END
$$
LANGUAGE plpgsql;
CREATE OR REPLACE FUNCTION getNearestRoadPlaceIdSlow(point GEOMETRY)
RETURNS BIGINT
AS $$
DECLARE
r RECORD;
search_diameter FLOAT;
BEGIN
search_diameter := 0.00005;
WHILE search_diameter < 0.1 LOOP
FOR r IN
SELECT place_id FROM placex
WHERE ST_DWithin(geometry, point, search_diameter)
AND rank_address BETWEEN 26 AND 27
ORDER BY ST_Distance(geometry, point) ASC limit 1
LOOP
RETURN r.place_id;
END LOOP;
search_diameter := search_diameter * 2;
END LOOP;
RETURN NULL;
END
$$
LANGUAGE plpgsql;
-- Tiger import function
CREATE OR REPLACE FUNCTION tiger_line_import(linegeo GEOMETRY, in_startnumber INTEGER, CREATE OR REPLACE FUNCTION tiger_line_import(linegeo GEOMETRY, in_startnumber INTEGER,
in_endnumber INTEGER, interpolationtype TEXT, in_endnumber INTEGER, interpolationtype TEXT,
token_info JSONB, in_postcode TEXT) RETURNS INTEGER token_info JSONB, in_postcode TEXT) RETURNS INTEGER
@@ -164,9 +71,6 @@ BEGIN
place_centroid := ST_Centroid(linegeo); place_centroid := ST_Centroid(linegeo);
out_partition := get_partition('us'); out_partition := get_partition('us');
-- HYBRID LOOKUP LOGIC (see gh-issue #2463)
-- if partition tables exist, use them for fast spatial lookups
{% if 'location_road_0' in db.tables %}
out_parent_place_id := getNearestNamedRoadPlaceId(out_partition, place_centroid, out_parent_place_id := getNearestNamedRoadPlaceId(out_partition, place_centroid,
token_info); token_info);
@@ -180,35 +84,15 @@ BEGIN
INTO out_parent_place_id; INTO out_parent_place_id;
END IF; END IF;
-- When updatable information has been dropped: --insert street(line) into import table
-- Partition tables no longer exist, but search_name still persists. insert into location_property_tiger_import (linegeo, place_id, partition,
{% elif 'search_name' in db.tables %}
-- Fallback: Look up in 'search_name' table
-- though spatial lookups here can be slower.
out_parent_place_id := getNearestNamedRoadPlaceIdSlow(place_centroid, token_info);
IF out_parent_place_id IS NULL THEN
out_parent_place_id := getNearestParallelRoadFeatureSlow(linegeo);
END IF;
IF out_parent_place_id IS NULL THEN
out_parent_place_id := getNearestRoadPlaceIdSlow(place_centroid);
END IF;
{% endif %}
-- If parent was found, insert street(line) into import table
IF out_parent_place_id IS NOT NULL THEN
INSERT INTO location_property_tiger_import (linegeo, place_id, partition,
parent_place_id, startnumber, endnumber, parent_place_id, startnumber, endnumber,
step, postcode) step, postcode)
VALUES (linegeo, nextval('seq_place'), out_partition, values (linegeo, nextval('seq_place'), out_partition,
out_parent_place_id, startnumber, endnumber, out_parent_place_id, startnumber, endnumber,
stepsize, in_postcode); stepsize, in_postcode);
RETURN 1; RETURN 1;
END IF;
RETURN 0;
END; END;
$$ $$
LANGUAGE plpgsql; LANGUAGE plpgsql;

View File

@@ -1,4 +1,4 @@
site_name: Nominatim Manual site_name: Nominatim 5.2.0 Manual
theme: theme:
font: false font: false
name: material name: material

View File

@@ -23,7 +23,7 @@ an ASGI-capable server like uvicorn. To install them from pypi run:
You need to have a Nominatim database imported with the 'nominatim-db' You need to have a Nominatim database imported with the 'nominatim-db'
package. Go to the project directory, then run uvicorn as: package. Go to the project directory, then run uvicorn as:
uvicorn --factory nominatim_api.server.falcon.server:run_wsgi uvicorn --factory nominatim.server.falcon.server:run_wsgi
## Documentation ## Documentation

View File

@@ -15,13 +15,12 @@ classifiers = [
"Operating System :: OS Independent", "Operating System :: OS Independent",
] ]
dependencies = [ dependencies = [
"psycopg != 3.3.0", "psycopg",
"python-dotenv", "python-dotenv",
"jinja2", "jinja2",
"pyYAML>=5.1", "pyYAML>=5.1",
"psutil", "psutil",
"PyICU", "PyICU"
"mwparserfromhell"
] ]
dynamic = ["version"] dynamic = ["version"]

View File

@@ -1,4 +1,3 @@
name: name:
default: الأراضي الفلسطينية default: Palestinian Territory
en: Palestinian Territories
"no": Det palestinske området "no": Det palestinske området

View File

@@ -305,7 +305,6 @@ ch:
names: !include country-names/ch.yaml names: !include country-names/ch.yaml
postcode: postcode:
pattern: "dddd" pattern: "dddd"
extent: 3000
# Côte d'Ivoire (Côte dIvoire) # Côte d'Ivoire (Côte dIvoire)
@@ -473,7 +472,6 @@ ee:
names: !include country-names/ee.yaml names: !include country-names/ee.yaml
postcode: postcode:
pattern: "ddddd" pattern: "ddddd"
extent: 3000
# Egypt (مصر) # Egypt (مصر)
@@ -587,7 +585,6 @@ gb:
postcode: postcode:
pattern: "(l?ld[A-Z0-9]?) ?(dll)" pattern: "(l?ld[A-Z0-9]?) ?(dll)"
output: \1 \2 output: \1 \2
extent: 700
# Grenada (Grenada) # Grenada (Grenada)
@@ -615,7 +612,6 @@ gg:
postcode: postcode:
pattern: "(GYdd?) ?(dll)" pattern: "(GYdd?) ?(dll)"
output: \1 \2 output: \1 \2
extent: 1000
# Ghana (Ghana) # Ghana (Ghana)
@@ -770,7 +766,6 @@ ie:
postcode: postcode:
pattern: "(ldd) ?([0123456789ACDEFHKNPRTVWXY]{4})" pattern: "(ldd) ?([0123456789ACDEFHKNPRTVWXY]{4})"
output: \1 \2 output: \1 \2
extent: 50
# Israel (ישראל) # Israel (ישראל)
@@ -790,7 +785,6 @@ im:
postcode: postcode:
pattern: "(IMdd?) ?(dll)" pattern: "(IMdd?) ?(dll)"
output: \1 \2 output: \1 \2
extent: 700
# India (India) # India (India)
@@ -885,7 +879,6 @@ jp:
postcode: postcode:
pattern: "(ddd)-?(dddd)" pattern: "(ddd)-?(dddd)"
output: \1-\2 output: \1-\2
extent: 3000
# Kenya (Kenya) # Kenya (Kenya)
@@ -1020,7 +1013,6 @@ li:
names: !include country-names/li.yaml names: !include country-names/li.yaml
postcode: postcode:
pattern: "dddd" pattern: "dddd"
extent: 4000
# Sri Lanka (ශ්‍රී ලංකාව இலங்கை) # Sri Lanka (ශ්‍රී ලංකාව இலங்கை)
@@ -1066,7 +1058,6 @@ lu:
names: !include country-names/lu.yaml names: !include country-names/lu.yaml
postcode: postcode:
pattern: "dddd" pattern: "dddd"
extent: 1000
# Latvia (Latvija) # Latvia (Latvija)
@@ -1299,7 +1290,6 @@ nl:
postcode: postcode:
pattern: "(dddd) ?(ll)" pattern: "(dddd) ?(ll)"
output: \1 \2 output: \1 \2
extent: 800
# Norway (Norge) # Norway (Norge)
@@ -1435,7 +1425,6 @@ pt:
names: !include country-names/pt.yaml names: !include country-names/pt.yaml
postcode: postcode:
pattern: "dddd(?:-ddd)?" pattern: "dddd(?:-ddd)?"
extent: 1000
# Palau (Belau) # Palau (Belau)
@@ -1471,7 +1460,6 @@ ro:
names: !include country-names/ro.yaml names: !include country-names/ro.yaml
postcode: postcode:
pattern: "dddddd" pattern: "dddddd"
extent: 2500
# Serbia (Србија) # Serbia (Србија)

View File

@@ -8,6 +8,7 @@
Helper functions for localizing names of results. Helper functions for localizing names of results.
""" """
from typing import Mapping, List, Optional from typing import Mapping, List, Optional
from .config import Configuration
from .results import AddressLines, BaseResultT from .results import AddressLines, BaseResultT
import re import re
@@ -17,15 +18,15 @@ class Locales:
""" Helper class for localization of names. """ Helper class for localization of names.
It takes a list of language prefixes in their order of preferred It takes a list of language prefixes in their order of preferred
usage and comma separated name keys (Configuration.OUTPUT_NAMES). usage.
""" """
def __init__(self, langs: Optional[List[str]] = None, def __init__(self, langs: Optional[List[str]] = None):
names: str = 'name:XX,name') -> None: self.config = Configuration(None)
self.languages = langs or [] self.languages = langs or []
self.name_tags: List[str] = [] self.name_tags: List[str] = []
parts = names.split(',') if names else [] parts = self.config.OUTPUT_NAMES.split(',')
for part in parts: for part in parts:
part = part.strip() part = part.strip()
@@ -67,7 +68,7 @@ class Locales:
return next(iter(names.values())) return next(iter(names.values()))
@staticmethod @staticmethod
def from_accept_languages(langstr: str, names: str = 'name:XX,name') -> 'Locales': def from_accept_languages(langstr: str) -> 'Locales':
""" Create a localization object from a language list in the """ Create a localization object from a language list in the
format of HTTP accept-languages header. format of HTTP accept-languages header.
@@ -95,7 +96,7 @@ class Locales:
if len(parts) > 1 and all(c[0] != parts[0] for c in candidates): if len(parts) > 1 and all(c[0] != parts[0] for c in candidates):
languages.append(parts[0]) languages.append(parts[0])
return Locales(languages, names) return Locales(languages)
def localize(self, lines: AddressLines) -> None: def localize(self, lines: AddressLines) -> None:
""" Sets the local name of address parts according to the chosen """ Sets the local name of address parts according to the chosen

View File

@@ -288,7 +288,7 @@ class TextLogger(BaseLogger):
self._write(f"rank={res.rank_address}, ") self._write(f"rank={res.rank_address}, ")
self._write(f"osm={''.join(map(str, res.osm_object or []))}, ") self._write(f"osm={''.join(map(str, res.osm_object or []))}, ")
self._write(f'cc={res.country_code}, ') self._write(f'cc={res.country_code}, ')
self._write(f'importance={res.importance or float("NaN"):.5f})\n') self._write(f'importance={res.importance or -1:.5f})\n')
total += 1 total += 1
self._write(f'TOTAL: {total}\n\n') self._write(f'TOTAL: {total}\n\n')

View File

@@ -2,7 +2,7 @@
# #
# This file is part of Nominatim. (https://nominatim.org) # This file is part of Nominatim. (https://nominatim.org)
# #
# Copyright (C) 2025 by the Nominatim developer community. # Copyright (C) 2024 by the Nominatim developer community.
# For a full list of authors see the git log. # For a full list of authors see the git log.
""" """
Implementation of place lookup by ID (doing many places at once). Implementation of place lookup by ID (doing many places at once).
@@ -291,30 +291,12 @@ async def find_in_postcode(conn: SearchConnection, collector: Collector) -> bool
.table_valued(sa.column('value', type_=sa.JSON)) .table_valued(sa.column('value', type_=sa.JSON))
t = conn.t.postcode t = conn.t.postcode
sql = sa.select(pid_tab.c.value['i'].as_integer().label('_idx'), sql = sa.select(pid_tab.c.value['i'].as_integer().label('_idx'),
t.c.osm_id, t.c.place_id, t.c.parent_place_id, t.c.place_id, t.c.parent_place_id,
t.c.rank_search, t.c.rank_search, t.c.rank_address,
t.c.indexed_date, t.c.postcode, t.c.country_code, t.c.indexed_date, t.c.postcode, t.c.country_code,
t.c.centroid)\ t.c.geometry.label('centroid'))\
.where(t.c.place_id == pid_tab.c.value['id'].as_string().cast(sa.BigInteger)) .where(t.c.place_id == pid_tab.c.value['id'].as_string().cast(sa.BigInteger))
if await collector.add_rows_from_sql(conn, sql, t.c.geometry,
nres.create_from_postcode_row):
return True
osm_ids = [{'i': i, 'oi': p.osm_id}
for i, p in collector.enumerate_free_osm_ids() if p.osm_type == 'R']
if osm_ids:
pid_tab = sa.func.JsonArrayEach(sa.type_coerce(osm_ids, sa.JSON))\
.table_valued(sa.column('value', type_=sa.JSON))
t = conn.t.postcode
sql = sa.select(pid_tab.c.value['i'].as_integer().label('_idx'),
t.c.osm_id, t.c.place_id, t.c.parent_place_id,
t.c.rank_search,
t.c.indexed_date, t.c.postcode, t.c.country_code,
t.c.centroid)\
.where(t.c.osm_id == pid_tab.c.value['oi'].as_string().cast(sa.BigInteger))
return await collector.add_rows_from_sql(conn, sql, t.c.geometry, return await collector.add_rows_from_sql(conn, sql, t.c.geometry,
nres.create_from_postcode_row) nres.create_from_postcode_row)

View File

@@ -10,7 +10,7 @@ Helper classes and functions for formatting results into API responses.
from typing import Type, TypeVar, Dict, List, Callable, Any, Mapping, Optional, cast from typing import Type, TypeVar, Dict, List, Callable, Any, Mapping, Optional, cast
from collections import defaultdict from collections import defaultdict
from pathlib import Path from pathlib import Path
import importlib.util import importlib
from .server.content_types import CONTENT_JSON from .server.content_types import CONTENT_JSON
@@ -43,7 +43,7 @@ class FormatDispatcher:
return decorator return decorator
def error_format_func(self, func: ErrorFormatFunc) -> ErrorFormatFunc: def error_format_func(self, func: ErrorFormatFunc) -> ErrorFormatFunc:
""" Decorator for a function that formats error messages. """ Decorator for a function that formats error messges.
There is only one error formatter per dispatcher. Using There is only one error formatter per dispatcher. Using
the decorator repeatedly will overwrite previous functions. the decorator repeatedly will overwrite previous functions.
""" """
@@ -79,7 +79,7 @@ class FormatDispatcher:
def set_content_type(self, fmt: str, content_type: str) -> None: def set_content_type(self, fmt: str, content_type: str) -> None:
""" Set the content type for the given format. This is the string """ Set the content type for the given format. This is the string
that will be returned in the Content-Type header of the HTML that will be returned in the Content-Type header of the HTML
response, when the given format is chosen. response, when the given format is choosen.
""" """
self.content_types[fmt] = content_type self.content_types[fmt] = content_type

View File

@@ -2,7 +2,7 @@
# #
# This file is part of Nominatim. (https://nominatim.org) # This file is part of Nominatim. (https://nominatim.org)
# #
# Copyright (C) 2025 by the Nominatim developer community. # Copyright (C) 2024 by the Nominatim developer community.
# For a full list of authors see the git log. # For a full list of authors see the git log.
""" """
Dataclasses for search results and helper functions to fill them. Dataclasses for search results and helper functions to fill them.
@@ -407,13 +407,11 @@ def create_from_postcode_row(row: SaRow, class_type: Type[BaseResultT]) -> BaseR
""" """
return class_type(source_table=SourceTable.POSTCODE, return class_type(source_table=SourceTable.POSTCODE,
place_id=row.place_id, place_id=row.place_id,
osm_object=None if row.osm_id is None else ('R', row.osm_id),
parent_place_id=row.parent_place_id, parent_place_id=row.parent_place_id,
category=(('place', 'postcode') if row.osm_id is None category=('place', 'postcode'),
else ('boundary', 'postal_code')),
names={'ref': row.postcode}, names={'ref': row.postcode},
rank_search=row.rank_search, rank_search=row.rank_search,
rank_address=5, rank_address=row.rank_address,
country_code=row.country_code, country_code=row.country_code,
centroid=Point.from_wkb(row.centroid), centroid=Point.from_wkb(row.centroid),
geometry=_filter_geometries(row)) geometry=_filter_geometries(row))
@@ -496,15 +494,17 @@ def _get_address_lookup_id(result: BaseResultT) -> int:
async def _finalize_entry(conn: SearchConnection, result: BaseResultT) -> None: async def _finalize_entry(conn: SearchConnection, result: BaseResultT) -> None:
assert result.address_rows is not None assert result.address_rows is not None
if result.category[0] not in ('boundary', 'place')\
postcode = result.postcode or (result.address and result.address.get('postcode')) or result.category[1] not in ('postal_code', 'postcode'):
postcode = result.postcode
if not postcode and result.address:
postcode = result.address.get('postcode')
if postcode and ',' not in postcode and ';' not in postcode: if postcode and ',' not in postcode and ';' not in postcode:
result.address_rows.append(AddressLine( result.address_rows.append(AddressLine(
category=('place', 'postcode'), category=('place', 'postcode'),
names={'ref': postcode}, names={'ref': postcode},
fromarea=False, isaddress=True, rank_address=5, fromarea=False, isaddress=True, rank_address=5,
distance=0.0)) distance=0.0))
if result.country_code: if result.country_code:
async def _get_country_names() -> Optional[Dict[str, str]]: async def _get_country_names() -> Optional[Dict[str, str]]:
t = conn.t.country_name t = conn.t.country_name
@@ -627,6 +627,13 @@ async def complete_address_details(conn: SearchConnection, results: List[BaseRes
if current_result.country_code is None and row.country_code: if current_result.country_code is None and row.country_code:
current_result.country_code = row.country_code current_result.country_code = row.country_code
if row.type in ('postcode', 'postal_code') and location_isaddress:
if not row.fromarea or \
(current_result.address and 'postcode' in current_result.address):
location_isaddress = False
else:
current_result.postcode = None
assert current_result.address_rows is not None assert current_result.address_rows is not None
current_result.address_rows.append(_result_row_to_address_row(row, location_isaddress)) current_result.address_rows.append(_result_row_to_address_row(row, location_isaddress))
current_rank_address = row.rank_address current_rank_address = row.rank_address

View File

@@ -157,19 +157,16 @@ class ReverseGeocoder:
include.extend(('natural', 'water', 'waterway')) include.extend(('natural', 'water', 'waterway'))
return table.c.class_.in_(tuple(include)) return table.c.class_.in_(tuple(include))
async def _find_closest_street_or_pois(self, distance: float, async def _find_closest_street_or_poi(self, distance: float) -> Optional[SaRow]:
fuzziness: float) -> list[SaRow]: """ Look up the closest rank 26+ place in the database, which
""" Look up the closest rank 26+ place in the database. is closer than the given distance.
The function finds the object that is closest to the reverse
search point as well as all objects within 'fuzziness' distance
to that best result.
""" """
t = self.conn.t.placex t = self.conn.t.placex
# PostgreSQL must not get the distance as a parameter because # PostgreSQL must not get the distance as a parameter because
# there is a danger it won't be able to properly estimate index use # there is a danger it won't be able to properly estimate index use
# when used with prepared statements # when used with prepared statements
diststr = sa.text(f"{distance + fuzziness}") diststr = sa.text(f"{distance}")
sql: SaLambdaSelect = sa.lambda_stmt( sql: SaLambdaSelect = sa.lambda_stmt(
lambda: _select_from_placex(t) lambda: _select_from_placex(t)
@@ -177,7 +174,9 @@ class ReverseGeocoder:
.where(t.c.indexed_status == 0) .where(t.c.indexed_status == 0)
.where(t.c.linked_place_id == None) .where(t.c.linked_place_id == None)
.where(sa.or_(sa.not_(t.c.geometry.is_area()), .where(sa.or_(sa.not_(t.c.geometry.is_area()),
t.c.centroid.ST_Distance(WKT_PARAM) < diststr))) t.c.centroid.ST_Distance(WKT_PARAM) < diststr))
.order_by('distance')
.limit(2))
if self.has_geometries(): if self.has_geometries():
sql = self._add_geometry_columns(sql, t.c.geometry) sql = self._add_geometry_columns(sql, t.c.geometry)
@@ -199,44 +198,24 @@ class ReverseGeocoder:
self._filter_by_layer(t))) self._filter_by_layer(t)))
if not restrict: if not restrict:
return [] return None
inner = sql.where(sa.or_(*restrict)) \ sql = sql.where(sa.or_(*restrict))
.add_columns(t.c.geometry.label('_geometry')) \
.subquery()
# Use a window function to get the closest results to the best result. # If the closest object is inside an area, then check if there is a
windowed = sa.select(inner, # POI node nearby and return that.
sa.func.first_value(inner.c.distance) prev_row = None
.over(order_by=inner.c.distance) for row in await self.conn.execute(sql, self.bind_params):
.label('_min_distance'), if prev_row is None:
sa.func.first_value( if row.rank_search <= 27 or row.osm_type == 'N' or row.distance > 0:
sa.case((inner.c.rank_search <= 27, return row
inner.c._geometry.ST_ClosestPoint(WKT_PARAM)), prev_row = row
else_=None)) else:
.over(order_by=inner.c.distance) if row.rank_search > 27 and row.osm_type == 'N'\
.label('_closest_point'), and row.distance < 0.0001:
sa.func.first_value(sa.case((sa.or_(inner.c.rank_search <= 27, return row
inner.c.osm_type == 'N'), None),
else_=inner.c._geometry))
.over(order_by=inner.c.distance)
.label('_best_geometry')) \
.subquery()
outer = sa.select(*(c for c in windowed.c if not c.key.startswith('_')), return prev_row
sa.case((sa.or_(windowed.c._closest_point == None,
windowed.c.housenumber == None), None),
else_=windowed.c.centroid.ST_Distance(windowed.c._closest_point))
.label('distance_from_best'),
sa.case((sa.or_(windowed.c._best_geometry == None,
windowed.c.rank_search <= 27,
windowed.c.osm_type != 'N'), False),
else_=windowed.c.centroid.ST_CoveredBy(windowed.c._best_geometry))
.label('best_inside')) \
.where(windowed.c.distance < windowed.c._min_distance + fuzziness) \
.order_by(windowed.c.distance)
return list(await self.conn.execute(outer, self.bind_params))
async def _find_housenumber_for_street(self, parent_place_id: int) -> Optional[SaRow]: async def _find_housenumber_for_street(self, parent_place_id: int) -> Optional[SaRow]:
t = self.conn.t.placex t = self.conn.t.placex
@@ -322,48 +301,32 @@ class ReverseGeocoder:
""" Find a street or POI/address for the given WKT point. """ Find a street or POI/address for the given WKT point.
""" """
log().section('Reverse lookup on street/address level') log().section('Reverse lookup on street/address level')
row_func: RowFunc = nres.create_from_placex_row
distance = 0.006 distance = 0.006
parent_place_id = None
result = None row = await self._find_closest_street_or_poi(distance)
hnr_distance = None row_func: RowFunc = nres.create_from_placex_row
parent_street = None log().var_dump('Result (street/building)', row)
for row in await self._find_closest_street_or_pois(distance, 0.001):
if result is None: # If the closest result was a street, but an address was requested,
log().var_dump('Closest result', row) # check for a housenumber nearby which is part of the street.
result = row if row is not None:
if self.max_rank > 27 \ if self.max_rank > 27 \
and self.layer_enabled(DataLayer.ADDRESS) \ and self.layer_enabled(DataLayer.ADDRESS) \
and result.rank_address <= 27: and row.rank_address <= 27:
parent_street = result.place_id
distance = 0.001 distance = 0.001
else: parent_place_id = row.place_id
distance = row.distance log().comment('Find housenumber for street')
# If the closest result was a street but an address was requested, addr_row = await self._find_housenumber_for_street(parent_place_id)
# see if we can refine the result with a housenumber closeby. log().var_dump('Result (street housenumber)', addr_row)
elif parent_street is not None \
and row.distance_from_best is not None \
and row.distance_from_best < 0.001 \
and (hnr_distance is None or hnr_distance > row.distance_from_best) \
and row.parent_place_id == parent_street:
log().var_dump('Housenumber to closest result', row)
result = row
hnr_distance = row.distance_from_best
distance = row.distance
# If the closest object is inside an area, then check if there is
# a POI nearby and return that with preference.
elif result.osm_type != 'N' and result.rank_search > 27 \
and result.distance == 0 \
and row.best_inside:
log().var_dump('POI near closest result area', row)
result = row
break # it can't get better than that, everything else is farther away
# For the US also check the TIGER data, when no housenumber/POI was found. if addr_row is not None:
if result is not None and parent_street is not None and hnr_distance is None \ row = addr_row
and result.country_code == 'us': row_func = nres.create_from_placex_row
distance = addr_row.distance
elif row.country_code == 'us' and parent_place_id is not None:
log().comment('Find TIGER housenumber for street') log().comment('Find TIGER housenumber for street')
addr_row = await self._find_tiger_number_for_street(parent_street) addr_row = await self._find_tiger_number_for_street(parent_place_id)
log().var_dump('Result (street Tiger housenumber)', addr_row) log().var_dump('Result (street Tiger housenumber)', addr_row)
if addr_row is not None: if addr_row is not None:
@@ -371,20 +334,22 @@ class ReverseGeocoder:
functools.partial(nres.create_from_tiger_row, functools.partial(nres.create_from_tiger_row,
osm_type=row.osm_type, osm_type=row.osm_type,
osm_id=row.osm_id)) osm_id=row.osm_id))
result = addr_row row = addr_row
else:
distance = row.distance
# Check for an interpolation that is either closer than our result # Check for an interpolation that is either closer than our result
# or belongs to a close street found. # or belongs to a close street found.
# No point in doing this when the result is already inside a building, if self.max_rank > 27 and self.layer_enabled(DataLayer.ADDRESS):
# i.e. when the distance is already 0.
if self.max_rank > 27 and self.layer_enabled(DataLayer.ADDRESS) and distance > 0:
log().comment('Find interpolation for street') log().comment('Find interpolation for street')
addr_row = await self._find_interpolation_for_street(parent_street, distance) addr_row = await self._find_interpolation_for_street(parent_place_id,
distance)
log().var_dump('Result (street interpolation)', addr_row) log().var_dump('Result (street interpolation)', addr_row)
if addr_row is not None: if addr_row is not None:
return addr_row, nres.create_from_osmline_row row = addr_row
row_func = nres.create_from_osmline_row
return result, row_func return row, row_func
async def _lookup_area_address(self) -> Optional[SaRow]: async def _lookup_area_address(self) -> Optional[SaRow]:
""" Lookup large addressable areas for the given WKT point. """ Lookup large addressable areas for the given WKT point.

View File

@@ -374,7 +374,7 @@ class SearchBuilder:
tokens = self.get_country_tokens(assignment.country) tokens = self.get_country_tokens(assignment.country)
if not tokens: if not tokens:
return None return None
sdata.set_countries(tokens) sdata.set_strings('countries', tokens)
sdata.penalty += self.query.get_in_word_penalty(assignment.country) sdata.penalty += self.query.get_in_word_penalty(assignment.country)
elif self.details.countries: elif self.details.countries:
sdata.countries = dbf.WeightedStrings(self.details.countries, sdata.countries = dbf.WeightedStrings(self.details.countries,
@@ -413,7 +413,7 @@ class SearchBuilder:
""" """
tokens = self.query.get_tokens(trange, qmod.TOKEN_COUNTRY) tokens = self.query.get_tokens(trange, qmod.TOKEN_COUNTRY)
if self.details.countries: if self.details.countries:
tokens = [t for t in tokens if t.get_country() in self.details.countries] tokens = [t for t in tokens if t.lookup_word in self.details.countries]
return tokens return tokens

View File

@@ -22,7 +22,7 @@ class CountedTokenIDs:
""" A list of token IDs with their respective counts, sorted """ A list of token IDs with their respective counts, sorted
from least frequent to most frequent. from least frequent to most frequent.
If a token count is one, then statistics are likely to be unavailable If a token count is one, then statistics are likely to be unavaible
and a relatively high count is assumed instead. and a relatively high count is assumed instead.
""" """
@@ -244,21 +244,6 @@ class SearchData:
setattr(self, field, wstrs) setattr(self, field, wstrs)
def set_countries(self, tokens: List[Token]) -> None:
""" Set the WeightedStrings properties for countries. Multiple
entries for the same country are deduplicated and the minimum
penalty is used. Adapts the global penalty, so that the
minimum penalty is 0.
"""
if tokens:
min_penalty = min(t.penalty for t in tokens)
self.penalty += min_penalty
countries: dict[str, float] = {}
for t in tokens:
cc = t.get_country()
countries[cc] = min(t.penalty - min_penalty, countries.get(cc, 10000))
self.countries = WeightedStrings(list(countries.keys()), list(countries.values()))
def set_qualifiers(self, tokens: List[Token]) -> None: def set_qualifiers(self, tokens: List[Token]) -> None:
""" Set the qulaifier field from the given tokens. """ Set the qulaifier field from the given tokens.
""" """

View File

@@ -175,8 +175,7 @@ class AddressSearch(base.AbstractSearch):
sql = sql.where(sa.select(tpc.c.postcode) sql = sql.where(sa.select(tpc.c.postcode)
.where(tpc.c.postcode.in_(self.postcodes.values)) .where(tpc.c.postcode.in_(self.postcodes.values))
.where(tpc.c.country_code == t.c.country_code) .where(tpc.c.country_code == t.c.country_code)
.where(t.c.centroid.intersects(tpc.c.geometry, .where(t.c.centroid.within_distance(tpc.c.geometry, 0.4))
use_index=False))
.exists()) .exists())
if details.viewbox is not None: if details.viewbox is not None:
@@ -226,7 +225,7 @@ class AddressSearch(base.AbstractSearch):
tpc = conn.t.postcode tpc = conn.t.postcode
pcs = self.postcodes.values pcs = self.postcodes.values
pc_near = sa.select(sa.func.min(tpc.c.centroid.ST_Distance(t.c.centroid) pc_near = sa.select(sa.func.min(tpc.c.geometry.ST_Distance(t.c.centroid)
* (tpc.c.rank_search - 19)))\ * (tpc.c.rank_search - 19)))\
.where(tpc.c.postcode.in_(pcs))\ .where(tpc.c.postcode.in_(pcs))\
.where(tpc.c.country_code == t.c.country_code)\ .where(tpc.c.country_code == t.c.country_code)\

View File

@@ -79,8 +79,7 @@ class PlaceSearch(base.AbstractSearch):
tpc = conn.t.postcode tpc = conn.t.postcode
sql = sql.where(sa.select(tpc.c.postcode) sql = sql.where(sa.select(tpc.c.postcode)
.where(tpc.c.postcode.in_(self.postcodes.values)) .where(tpc.c.postcode.in_(self.postcodes.values))
.where(t.c.centroid.intersects(tpc.c.geometry, .where(t.c.centroid.within_distance(tpc.c.geometry, 0.4))
use_index=False))
.exists()) .exists())
if details.viewbox is not None: if details.viewbox is not None:
@@ -158,7 +157,7 @@ class PlaceSearch(base.AbstractSearch):
tpc = conn.t.postcode tpc = conn.t.postcode
pcs = self.postcodes.values pcs = self.postcodes.values
pc_near = sa.select(sa.func.min(tpc.c.centroid.ST_Distance(t.c.centroid)))\ pc_near = sa.select(sa.func.min(tpc.c.geometry.ST_Distance(t.c.centroid)))\
.where(tpc.c.postcode.in_(pcs))\ .where(tpc.c.postcode.in_(pcs))\
.scalar_subquery() .scalar_subquery()
penalty += sa.case((t.c.postcode.in_(pcs), 0.0), penalty += sa.case((t.c.postcode.in_(pcs), 0.0),

View File

@@ -14,7 +14,7 @@ from . import base
from ...typing import SaBind, SaExpression from ...typing import SaBind, SaExpression
from ...sql.sqlalchemy_types import Geometry, IntArray from ...sql.sqlalchemy_types import Geometry, IntArray
from ...connection import SearchConnection from ...connection import SearchConnection
from ...types import SearchDetails from ...types import SearchDetails, Bbox
from ... import results as nres from ... import results as nres
from ..db_search_fields import SearchData from ..db_search_fields import SearchData
@@ -42,9 +42,10 @@ class PostcodeSearch(base.AbstractSearch):
t = conn.t.postcode t = conn.t.postcode
pcs = self.postcodes.values pcs = self.postcodes.values
sql = sa.select(t.c.place_id, t.c.parent_place_id, t.c.osm_id, sql = sa.select(t.c.place_id, t.c.parent_place_id,
t.c.rank_search, t.c.postcode, t.c.country_code, t.c.rank_search, t.c.rank_address,
t.c.centroid)\ t.c.postcode, t.c.country_code,
t.c.geometry.label('centroid'))\
.where(t.c.postcode.in_(pcs)) .where(t.c.postcode.in_(pcs))
if details.geometry_output: if details.geometry_output:
@@ -58,7 +59,7 @@ class PostcodeSearch(base.AbstractSearch):
else_=1.0) else_=1.0)
if details.near is not None: if details.near is not None:
sql = sql.order_by(t.c.centroid.ST_Distance(NEAR_PARAM)) sql = sql.order_by(t.c.geometry.ST_Distance(NEAR_PARAM))
sql = base.filter_by_area(sql, t, details) sql = base.filter_by_area(sql, t, details)
@@ -99,8 +100,28 @@ class PostcodeSearch(base.AbstractSearch):
results = nres.SearchResults() results = nres.SearchResults()
for row in await conn.execute(sql, bind_params): for row in await conn.execute(sql, bind_params):
p = conn.t.placex
placex_sql = base.select_placex(p)\
.add_columns(p.c.importance)\
.where(sa.text("""class = 'boundary'
AND type = 'postal_code'
AND osm_type = 'R'"""))\
.where(p.c.country_code == row.country_code)\
.where(p.c.postcode == row.postcode)\
.limit(1)
if details.geometry_output:
placex_sql = base.add_geometry_columns(placex_sql, p.c.geometry, details)
for prow in await conn.execute(placex_sql, bind_params):
result = nres.create_from_placex_row(prow, nres.SearchResult)
if result is not None:
result.bbox = Bbox.from_wkb(prow.bbox)
break
else:
result = nres.create_from_postcode_row(row, nres.SearchResult) result = nres.create_from_postcode_row(row, nres.SearchResult)
if result.place_id not in details.excluded:
result.accuracy = row.accuracy result.accuracy = row.accuracy
results.append(result) results.append(result)

View File

@@ -59,16 +59,12 @@ class ICUToken(qmod.Token):
assert self.info assert self.info
return self.info.get('class', ''), self.info.get('type', '') return self.info.get('class', ''), self.info.get('type', '')
def get_country(self) -> str: def rematch(self, norm: str) -> None:
assert self.info
return cast(str, self.info.get('cc', ''))
def match_penalty(self, norm: str) -> float:
""" Check how well the token matches the given normalized string """ Check how well the token matches the given normalized string
and add a penalty, if necessary. and add a penalty, if necessary.
""" """
if not self.lookup_word: if not self.lookup_word:
return 0.0 return
seq = difflib.SequenceMatcher(a=self.lookup_word, b=norm) seq = difflib.SequenceMatcher(a=self.lookup_word, b=norm)
distance = 0 distance = 0
@@ -79,7 +75,7 @@ class ICUToken(qmod.Token):
distance += max((ato-afrom), (bto-bfrom)) distance += max((ato-afrom), (bto-bfrom))
elif tag != 'equal': elif tag != 'equal':
distance += abs((ato-afrom) - (bto-bfrom)) distance += abs((ato-afrom) - (bto-bfrom))
return (distance/len(self.lookup_word)) self.penalty += (distance/len(self.lookup_word))
@staticmethod @staticmethod
def from_db_row(row: SaRow) -> 'ICUToken': def from_db_row(row: SaRow) -> 'ICUToken':
@@ -334,10 +330,9 @@ class ICUQueryAnalyzer(AbstractQueryAnalyzer):
norm = ''.join(f"{n.term_normalized}{'' if n.btype == qmod.BREAK_TOKEN else ' '}" norm = ''.join(f"{n.term_normalized}{'' if n.btype == qmod.BREAK_TOKEN else ' '}"
for n in query.nodes[start + 1:end + 1]).strip() for n in query.nodes[start + 1:end + 1]).strip()
for ttype, tokens in tlist.items(): for ttype, tokens in tlist.items():
if ttype != qmod.TOKEN_COUNTRY:
for token in tokens: for token in tokens:
itok = cast(ICUToken, token) cast(ICUToken, token).rematch(norm)
itok.penalty += itok.match_penalty(norm) * \
(1 if ttype in (qmod.TOKEN_WORD, qmod.TOKEN_PARTIAL) else 2)
def compute_break_penalties(self, query: qmod.QueryStruct) -> None: def compute_break_penalties(self, query: qmod.QueryStruct) -> None:
""" Set the break penalties for the nodes in the query. """ Set the break penalties for the nodes in the query.

View File

@@ -17,7 +17,7 @@ import dataclasses
# The x value for the regression computation will be the position of the # The x value for the regression computation will be the position of the
# token in the query. Thus we know the x values will be [0, query length). # token in the query. Thus we know the x values will be [0, query length).
# As the denominator only depends on the x values, we can pre-compute here # As the denominator only depends on the x values, we can pre-compute here
# the denominator to use for a given query length. # the denominatior to use for a given query length.
# Note that query length of two or less is special cased and will not use # Note that query length of two or less is special cased and will not use
# the values from this array. Thus it is not a problem that they are 0. # the values from this array. Thus it is not a problem that they are 0.
LINFAC = [i * (sum(si * si for si in range(i)) - (i - 1) * i * (i - 1) / 4) LINFAC = [i * (sum(si * si for si in range(i)) - (i - 1) * i * (i - 1) / 4)
@@ -127,12 +127,6 @@ class Token(ABC):
category objects. category objects.
""" """
@abstractmethod
def get_country(self) -> str:
""" Return the country code this token is associated with
(currently for country tokens only).
"""
@dataclasses.dataclass @dataclasses.dataclass
class TokenRange: class TokenRange:
@@ -231,7 +225,7 @@ class QueryNode:
return max(0, -self.penalty) return max(0, -self.penalty)
def name_address_ratio(self) -> float: def name_address_ratio(self) -> float:
""" Return the probability that the partial token belonging to """ Return the propability that the partial token belonging to
this node forms part of a name (as opposed of part of the address). this node forms part of a name (as opposed of part of the address).
""" """
if self.partial is None: if self.partial is None:
@@ -275,7 +269,7 @@ class QueryStruct:
directed acyclic graph. directed acyclic graph.
A query also has a direction penalty 'dir_penalty'. This describes A query also has a direction penalty 'dir_penalty'. This describes
the likelihood if the query should be read from left-to-right or the likelyhood if the query should be read from left-to-right or
vice versa. A negative 'dir_penalty' should be read as a penalty on vice versa. A negative 'dir_penalty' should be read as a penalty on
right-to-left reading, while a positive value represents a penalty right-to-left reading, while a positive value represents a penalty
for left-to-right reading. The default value is 0, which is equivalent for left-to-right reading. The default value is 0, which is equivalent

View File

@@ -7,8 +7,6 @@
""" """
Server implementation using the falcon webserver framework. Server implementation using the falcon webserver framework.
""" """
from __future__ import annotations
from typing import Optional, Mapping, Any, List, cast from typing import Optional, Mapping, Any, List, cast
from pathlib import Path from pathlib import Path
import asyncio import asyncio
@@ -163,7 +161,7 @@ class APIMiddleware:
def __init__(self, project_dir: Path, environ: Optional[Mapping[str, str]]) -> None: def __init__(self, project_dir: Path, environ: Optional[Mapping[str, str]]) -> None:
self.api = NominatimAPIAsync(project_dir, environ) self.api = NominatimAPIAsync(project_dir, environ)
self.app: Optional[App[Request, Response]] = None self.app: Optional[App] = None
@property @property
def config(self) -> Configuration: def config(self) -> Configuration:
@@ -171,7 +169,7 @@ class APIMiddleware:
""" """
return self.api.config return self.api.config
def set_app(self, app: App[Request, Response]) -> None: def set_app(self, app: App) -> None:
""" Set the Falcon application this middleware is connected to. """ Set the Falcon application this middleware is connected to.
""" """
self.app = app self.app = app
@@ -195,7 +193,7 @@ class APIMiddleware:
def get_application(project_dir: Path, def get_application(project_dir: Path,
environ: Optional[Mapping[str, str]] = None) -> App[Request, Response]: environ: Optional[Mapping[str, str]] = None) -> App:
""" Create a Nominatim Falcon ASGI application. """ Create a Nominatim Falcon ASGI application.
""" """
apimw = APIMiddleware(project_dir, environ) apimw = APIMiddleware(project_dir, environ)
@@ -217,7 +215,7 @@ def get_application(project_dir: Path,
return app return app
def run_wsgi() -> App[Request, Response]: def run_wsgi() -> App:
""" Entry point for uvicorn. """ Entry point for uvicorn.
Make sure uvicorn is run from the project directory. Make sure uvicorn is run from the project directory.

View File

@@ -50,7 +50,7 @@ class ParamWrapper(ASGIAdaptor):
headers={'content-type': self.content_type}) headers={'content-type': self.content_type})
def create_response(self, status: int, output: str, num_results: int) -> Response: def create_response(self, status: int, output: str, num_results: int) -> Response:
setattr(self.request.state, 'num_results', num_results) self.request.state.num_results = num_results
return Response(output, status_code=status, media_type=self.content_type) return Response(output, status_code=status, media_type=self.content_type)
def base_uri(self) -> str: def base_uri(self) -> str:
@@ -95,7 +95,7 @@ class FileLoggingMiddleware(BaseHTTPMiddleware):
async def dispatch(self, request: Request, async def dispatch(self, request: Request,
call_next: RequestResponseEndpoint) -> Response: call_next: RequestResponseEndpoint) -> Response:
qs = QueryStatistics() qs = QueryStatistics()
setattr(request.state, 'query_stats', qs) request.state.query_stats = qs
response = await call_next(request) response = await call_next(request)
if response.status_code != 200 or 'start' not in qs: if response.status_code != 200 or 'start' not in qs:

View File

@@ -2,7 +2,7 @@
# #
# This file is part of Nominatim. (https://nominatim.org) # This file is part of Nominatim. (https://nominatim.org)
# #
# Copyright (C) 2025 by the Nominatim developer community. # Copyright (C) 2024 by the Nominatim developer community.
# For a full list of authors see the git log. # For a full list of authors see the git log.
""" """
Custom functions and expressions for SQLAlchemy. Custom functions and expressions for SQLAlchemy.
@@ -32,6 +32,7 @@ def _default_intersects(element: PlacexGeometryReverseLookuppolygon,
compiler: 'sa.Compiled', **kw: Any) -> str: compiler: 'sa.Compiled', **kw: Any) -> str:
return ("(ST_GeometryType(placex.geometry) in ('ST_Polygon', 'ST_MultiPolygon')" return ("(ST_GeometryType(placex.geometry) in ('ST_Polygon', 'ST_MultiPolygon')"
" AND placex.rank_address between 4 and 25" " AND placex.rank_address between 4 and 25"
" AND placex.type != 'postcode'"
" AND placex.name is not null" " AND placex.name is not null"
" AND placex.indexed_status = 0" " AND placex.indexed_status = 0"
" AND placex.linked_place_id is null)") " AND placex.linked_place_id is null)")
@@ -42,6 +43,7 @@ def _sqlite_intersects(element: PlacexGeometryReverseLookuppolygon,
compiler: 'sa.Compiled', **kw: Any) -> str: compiler: 'sa.Compiled', **kw: Any) -> str:
return ("(ST_GeometryType(placex.geometry) in ('POLYGON', 'MULTIPOLYGON')" return ("(ST_GeometryType(placex.geometry) in ('POLYGON', 'MULTIPOLYGON')"
" AND placex.rank_address between 4 and 25" " AND placex.rank_address between 4 and 25"
" AND placex.type != 'postcode'"
" AND placex.name is not null" " AND placex.name is not null"
" AND placex.indexed_status = 0" " AND placex.indexed_status = 0"
" AND placex.linked_place_id is null)") " AND placex.linked_place_id is null)")
@@ -62,6 +64,7 @@ def default_reverse_place_diameter(element: IntersectsReverseDistance,
compiler: 'sa.Compiled', **kw: Any) -> str: compiler: 'sa.Compiled', **kw: Any) -> str:
table = element.tablename table = element.tablename
return f"({table}.rank_address between 4 and 25"\ return f"({table}.rank_address between 4 and 25"\
f" AND {table}.type != 'postcode'"\
f" AND {table}.name is not null"\ f" AND {table}.name is not null"\
f" AND {table}.linked_place_id is null"\ f" AND {table}.linked_place_id is null"\
f" AND {table}.osm_type = 'N'" + \ f" AND {table}.osm_type = 'N'" + \
@@ -76,6 +79,7 @@ def sqlite_reverse_place_diameter(element: IntersectsReverseDistance,
table = element.tablename table = element.tablename
return (f"({table}.rank_address between 4 and 25" return (f"({table}.rank_address between 4 and 25"
f" AND {table}.type != 'postcode'"
f" AND {table}.name is not null" f" AND {table}.name is not null"
f" AND {table}.linked_place_id is null" f" AND {table}.linked_place_id is null"
f" AND {table}.osm_type = 'N'" f" AND {table}.osm_type = 'N'"

View File

@@ -2,7 +2,7 @@
# #
# This file is part of Nominatim. (https://nominatim.org) # This file is part of Nominatim. (https://nominatim.org)
# #
# Copyright (C) 2025 by the Nominatim developer community. # Copyright (C) 2024 by the Nominatim developer community.
# For a full list of authors see the git log. # For a full list of authors see the git log.
""" """
SQLAlchemy definitions for all tables used by the frontend. SQLAlchemy definitions for all tables used by the frontend.
@@ -67,16 +67,15 @@ class SearchTables:
sa.Column('isaddress', sa.Boolean)) sa.Column('isaddress', sa.Boolean))
self.postcode = sa.Table( self.postcode = sa.Table(
'location_postcodes', meta, 'location_postcode', meta,
sa.Column('place_id', sa.BigInteger), sa.Column('place_id', sa.BigInteger),
sa.Column('parent_place_id', sa.BigInteger), sa.Column('parent_place_id', sa.BigInteger),
sa.Column('osm_id', sa.BigInteger),
sa.Column('rank_search', sa.SmallInteger), sa.Column('rank_search', sa.SmallInteger),
sa.Column('rank_address', sa.SmallInteger),
sa.Column('indexed_status', sa.SmallInteger), sa.Column('indexed_status', sa.SmallInteger),
sa.Column('indexed_date', sa.DateTime), sa.Column('indexed_date', sa.DateTime),
sa.Column('country_code', sa.String(2)), sa.Column('country_code', sa.String(2)),
sa.Column('postcode', sa.Text), sa.Column('postcode', sa.Text),
sa.Column('centroid', Geometry),
sa.Column('geometry', Geometry)) sa.Column('geometry', Geometry))
self.osmline = sa.Table( self.osmline = sa.Table(

View File

@@ -2,7 +2,7 @@
# #
# This file is part of Nominatim. (https://nominatim.org) # This file is part of Nominatim. (https://nominatim.org)
# #
# Copyright (C) 2025 by the Nominatim developer community. # Copyright (C) 2024 by the Nominatim developer community.
# For a full list of authors see the git log. # For a full list of authors see the git log.
""" """
Custom types for SQLAlchemy. Custom types for SQLAlchemy.
@@ -178,8 +178,6 @@ class Geometry(types.UserDefinedType): # type: ignore[type-arg]
def bind_processor(self, dialect: 'sa.Dialect') -> Callable[[Any], str]: def bind_processor(self, dialect: 'sa.Dialect') -> Callable[[Any], str]:
def process(value: Any) -> str: def process(value: Any) -> str:
if value is None:
return 'null'
if isinstance(value, str): if isinstance(value, str):
return value return value

View File

@@ -2,7 +2,7 @@
# #
# This file is part of Nominatim. (https://nominatim.org) # This file is part of Nominatim. (https://nominatim.org)
# #
# Copyright (C) 2025 by the Nominatim developer community. # Copyright (C) 2024 by the Nominatim developer community.
# For a full list of authors see the git log. # For a full list of authors see the git log.
""" """
Hard-coded information about tag categories. Hard-coded information about tag categories.
@@ -20,9 +20,7 @@ def get_label_tag(category: Tuple[str, str], extratags: Optional[Mapping[str, st
rank: int, country: Optional[str]) -> str: rank: int, country: Optional[str]) -> str:
""" Create a label tag for the given place that can be used as an XML name. """ Create a label tag for the given place that can be used as an XML name.
""" """
if category in (('place', 'postcode'), ('boundary', 'postal_code')): if rank < 26 and extratags and 'place' in extratags:
label = 'postcode'
elif rank < 26 and extratags and 'place' in extratags:
label = extratags['place'] label = extratags['place']
elif rank < 26 and extratags and 'linked_place' in extratags: elif rank < 26 and extratags and 'linked_place' in extratags:
label = extratags['linked_place'] label = extratags['linked_place']
@@ -30,6 +28,8 @@ def get_label_tag(category: Tuple[str, str], extratags: Optional[Mapping[str, st
label = ADMIN_LABELS.get((country or '', rank // 2))\ label = ADMIN_LABELS.get((country or '', rank // 2))\
or ADMIN_LABELS.get(('', rank // 2))\ or ADMIN_LABELS.get(('', rank // 2))\
or 'Administrative' or 'Administrative'
elif category[1] == 'postal_code':
label = 'postcode'
elif rank < 26: elif rank < 26:
label = category[1] if category[1] != 'yes' else category[0] label = category[1] if category[1] != 'yes' else category[0]
elif rank < 28: elif rank < 28:

View File

@@ -132,7 +132,7 @@ def _add_parent_rows_grouped(writer: JsonWriter, rows: AddressLines,
@dispatch.format_func(DetailedResult, 'json') @dispatch.format_func(DetailedResult, 'json')
def _format_details_json(result: DetailedResult, options: Mapping[str, Any]) -> str: def _format_details_json(result: DetailedResult, options: Mapping[str, Any]) -> str:
locales = options.get('locales') or Locales() locales = options.get('locales', Locales())
geom = result.geometry.get('geojson') geom = result.geometry.get('geojson')
centroid = result.centroid.to_geojson() centroid = result.centroid.to_geojson()

View File

@@ -2,7 +2,7 @@
# #
# This file is part of Nominatim. (https://nominatim.org) # This file is part of Nominatim. (https://nominatim.org)
# #
# Copyright (C) 2025 by the Nominatim developer community. # Copyright (C) 2024 by the Nominatim developer community.
# For a full list of authors see the git log. # For a full list of authors see the git log.
""" """
Helper function for parsing parameters and and outputting data Helper function for parsing parameters and and outputting data
@@ -12,7 +12,7 @@ from typing import Tuple, Optional, Any, Dict, Iterable
from itertools import chain from itertools import chain
import re import re
from ..results import SearchResults, SourceTable from ..results import SearchResult, SearchResults, SourceTable
from ..types import SearchDetails, GeometryFormat from ..types import SearchDetails, GeometryFormat
@@ -106,6 +106,10 @@ def deduplicate_results(results: SearchResults, max_results: int) -> SearchResul
classification_done = set() classification_done = set()
deduped = SearchResults() deduped = SearchResults()
for result in results: for result in results:
if result.source_table == SourceTable.POSTCODE:
assert result.names and 'ref' in result.names
if any(_is_postcode_relation_for(r, result.names['ref']) for r in results):
continue
if result.source_table == SourceTable.PLACEX: if result.source_table == SourceTable.PLACEX:
classification = (result.osm_object[0] if result.osm_object else None, classification = (result.osm_object[0] if result.osm_object else None,
result.category, result.category,
@@ -124,6 +128,15 @@ def deduplicate_results(results: SearchResults, max_results: int) -> SearchResul
return deduped return deduped
def _is_postcode_relation_for(result: SearchResult, postcode: str) -> bool:
return result.source_table == SourceTable.PLACEX \
and result.osm_object is not None \
and result.osm_object[0] == 'R' \
and result.category == ('boundary', 'postal_code') \
and result.names is not None \
and result.names.get('ref') == postcode
def _deg(axis: str) -> str: def _deg(axis: str) -> str:
return f"(?P<{axis}_deg>\\d+\\.\\d+)°?" return f"(?P<{axis}_deg>\\d+\\.\\d+)°?"

View File

@@ -2,7 +2,7 @@
# #
# This file is part of Nominatim. (https://nominatim.org) # This file is part of Nominatim. (https://nominatim.org)
# #
# Copyright (C) 2026 by the Nominatim developer community. # Copyright (C) 2025 by the Nominatim developer community.
# For a full list of authors see the git log. # For a full list of authors see the git log.
""" """
Generic part of the server implementation of the v1 API. Generic part of the server implementation of the v1 API.
@@ -174,8 +174,7 @@ async def details_endpoint(api: NominatimAPIAsync, params: ASGIAdaptor) -> Any:
if result is None: if result is None:
params.raise_error('No place with that OSM ID found.', status=404) params.raise_error('No place with that OSM ID found.', status=404)
locales = Locales.from_accept_languages(get_accepted_languages(params), locales = Locales.from_accept_languages(get_accepted_languages(params))
params.config().OUTPUT_NAMES)
locales.localize_results([result]) locales.localize_results([result])
output = params.formatting().format_result( output = params.formatting().format_result(
@@ -200,7 +199,6 @@ async def reverse_endpoint(api: NominatimAPIAsync, params: ASGIAdaptor) -> Any:
details['max_rank'] = helpers.zoom_to_rank(params.get_int('zoom', 18)) details['max_rank'] = helpers.zoom_to_rank(params.get_int('zoom', 18))
details['layers'] = get_layers(params) details['layers'] = get_layers(params)
details['query_stats'] = params.query_stats() details['query_stats'] = params.query_stats()
details['entrances'] = params.get_bool('entrances', False)
result = await api.reverse(coord, **details) result = await api.reverse(coord, **details)
@@ -217,8 +215,8 @@ async def reverse_endpoint(api: NominatimAPIAsync, params: ASGIAdaptor) -> Any:
query = '' query = ''
if result: if result:
Locales.from_accept_languages(get_accepted_languages(params), Locales.from_accept_languages(get_accepted_languages(params)).localize_results(
params.config().OUTPUT_NAMES).localize_results([result]) [result])
fmt_options = {'query': query, fmt_options = {'query': query,
'extratags': params.get_bool('extratags', False), 'extratags': params.get_bool('extratags', False),
@@ -239,7 +237,6 @@ async def lookup_endpoint(api: NominatimAPIAsync, params: ASGIAdaptor) -> Any:
debug = setup_debugging(params) debug = setup_debugging(params)
details = parse_geometry_details(params, fmt) details = parse_geometry_details(params, fmt)
details['query_stats'] = params.query_stats() details['query_stats'] = params.query_stats()
details['entrances'] = params.get_bool('entrances', False)
places = [] places = []
for oid in (params.get('osm_ids') or '').split(','): for oid in (params.get('osm_ids') or '').split(','):
@@ -258,8 +255,7 @@ async def lookup_endpoint(api: NominatimAPIAsync, params: ASGIAdaptor) -> Any:
if debug: if debug:
return build_response(params, loglib.get_and_disable(), num_results=len(results)) return build_response(params, loglib.get_and_disable(), num_results=len(results))
Locales.from_accept_languages(get_accepted_languages(params), Locales.from_accept_languages(get_accepted_languages(params)).localize_results(results)
params.config().OUTPUT_NAMES).localize_results(results)
fmt_options = {'extratags': params.get_bool('extratags', False), fmt_options = {'extratags': params.get_bool('extratags', False),
'namedetails': params.get_bool('namedetails', False), 'namedetails': params.get_bool('namedetails', False),
@@ -352,8 +348,7 @@ async def search_endpoint(api: NominatimAPIAsync, params: ASGIAdaptor) -> Any:
except UsageError as err: except UsageError as err:
params.raise_error(str(err)) params.raise_error(str(err))
Locales.from_accept_languages(get_accepted_languages(params), Locales.from_accept_languages(get_accepted_languages(params)).localize_results(results)
params.config().OUTPUT_NAMES).localize_results(results)
if details['dedupe'] and len(results) > 1: if details['dedupe'] and len(results) > 1:
results = helpers.deduplicate_results(results, max_results) results = helpers.deduplicate_results(results, max_results)

View File

@@ -65,14 +65,14 @@ class UpdateAddData:
def run(self, args: NominatimArgs) -> int: def run(self, args: NominatimArgs) -> int:
from ..tools import add_osm_data from ..tools import add_osm_data
if args.tiger_data:
return asyncio.run(self._add_tiger_data(args))
with connect(args.config.get_libpq_dsn()) as conn: with connect(args.config.get_libpq_dsn()) as conn:
if is_frozen(conn): if is_frozen(conn):
print('Database is marked frozen. New data can\'t be added.') print('Database is marked frozen. New data can\'t be added.')
return 1 return 1
if args.tiger_data:
return asyncio.run(self._add_tiger_data(args))
osm2pgsql_params = args.osm2pgsql_options(default_cache=1000, default_threads=1) osm2pgsql_params = args.osm2pgsql_options(default_cache=1000, default_threads=1)
if args.file or args.diff: if args.file or args.diff:
return add_osm_data.add_data_from_file(args.config.get_libpq_dsn(), return add_osm_data.add_data_from_file(args.config.get_libpq_dsn(),

View File

@@ -19,7 +19,6 @@ import nominatim_api as napi
from nominatim_api.v1.helpers import zoom_to_rank, deduplicate_results from nominatim_api.v1.helpers import zoom_to_rank, deduplicate_results
from nominatim_api.server.content_types import CONTENT_JSON from nominatim_api.server.content_types import CONTENT_JSON
import nominatim_api.logging as loglib import nominatim_api.logging as loglib
from ..config import Configuration
from ..errors import UsageError from ..errors import UsageError
from .args import NominatimArgs from .args import NominatimArgs
@@ -92,19 +91,18 @@ def _get_geometry_output(args: NominatimArgs) -> napi.GeometryFormat:
raise UsageError(f"Unknown polygon output format '{args.polygon_output}'.") from exp raise UsageError(f"Unknown polygon output format '{args.polygon_output}'.") from exp
def _get_locales(args: NominatimArgs, config: Configuration) -> napi.Locales: def _get_locales(args: NominatimArgs, default: Optional[str]) -> napi.Locales:
""" Get the locales from the language parameter. """ Get the locales from the language parameter.
""" """
language = args.lang or config.DEFAULT_LANGUAGE if args.lang:
output_names = config.OUTPUT_NAMES return napi.Locales.from_accept_languages(args.lang)
if default:
if language: return napi.Locales.from_accept_languages(default)
return napi.Locales.from_accept_languages(language, output_names)
return napi.Locales() return napi.Locales()
def _get_layers(args: NominatimArgs, default: Optional[napi.DataLayer]) -> Optional[napi.DataLayer]: def _get_layers(args: NominatimArgs, default: napi.DataLayer) -> Optional[napi.DataLayer]:
""" Get the list of selected layers as a DataLayer enum. """ Get the list of selected layers as a DataLayer enum.
""" """
if not args.layers: if not args.layers:
@@ -136,7 +134,7 @@ def _print_output(formatter: napi.FormatDispatcher, result: Any,
json.dump(json.loads(output), sys.stdout, indent=4, ensure_ascii=False) json.dump(json.loads(output), sys.stdout, indent=4, ensure_ascii=False)
except json.decoder.JSONDecodeError as err: except json.decoder.JSONDecodeError as err:
# Catch the error here, so that data can be debugged, # Catch the error here, so that data can be debugged,
# when people are developing custom result formatters. # when people are developping custom result formatters.
LOG.fatal("Parsing json failed: %s\nUnformatted output:\n%s", err, output) LOG.fatal("Parsing json failed: %s\nUnformatted output:\n%s", err, output)
else: else:
sys.stdout.write(output) sys.stdout.write(output)
@@ -173,10 +171,6 @@ class APISearch:
help='Preferred area to find search results') help='Preferred area to find search results')
group.add_argument('--bounded', action='store_true', group.add_argument('--bounded', action='store_true',
help='Strictly restrict results to viewbox area') help='Strictly restrict results to viewbox area')
group.add_argument('--layer', metavar='LAYER',
choices=[n.name.lower() for n in napi.DataLayer if n.name],
action='append', required=False, dest='layers',
help='Restrict results to one or more layers (may be repeated)')
group.add_argument('--no-dedupe', action='store_false', dest='dedupe', group.add_argument('--no-dedupe', action='store_false', dest='dedupe',
help='Do not remove duplicates from the result list') help='Do not remove duplicates from the result list')
_add_list_format(parser) _add_list_format(parser)
@@ -193,8 +187,6 @@ class APISearch:
raise UsageError(f"Unsupported format '{args.format}'. " raise UsageError(f"Unsupported format '{args.format}'. "
'Use --list-formats to see supported formats.') 'Use --list-formats to see supported formats.')
layers = _get_layers(args, None)
try: try:
with napi.NominatimAPI(args.project_dir) as api: with napi.NominatimAPI(args.project_dir) as api:
params: Dict[str, Any] = {'max_results': args.limit + min(args.limit, 10), params: Dict[str, Any] = {'max_results': args.limit + min(args.limit, 10),
@@ -205,7 +197,6 @@ class APISearch:
'excluded': args.exclude_place_ids, 'excluded': args.exclude_place_ids,
'viewbox': args.viewbox, 'viewbox': args.viewbox,
'bounded_viewbox': args.bounded, 'bounded_viewbox': args.bounded,
'layers': layers,
'entrances': args.entrances, 'entrances': args.entrances,
} }
@@ -223,7 +214,7 @@ class APISearch:
except napi.UsageError as ex: except napi.UsageError as ex:
raise UsageError(ex) from ex raise UsageError(ex) from ex
locales = _get_locales(args, api.config) locales = _get_locales(args, api.config.DEFAULT_LANGUAGE)
locales.localize_results(results) locales.localize_results(results)
if args.dedupe and len(results) > 1: if args.dedupe and len(results) > 1:
@@ -262,7 +253,7 @@ class APIReverse:
group.add_argument('--layer', metavar='LAYER', group.add_argument('--layer', metavar='LAYER',
choices=[n.name.lower() for n in napi.DataLayer if n.name], choices=[n.name.lower() for n in napi.DataLayer if n.name],
action='append', required=False, dest='layers', action='append', required=False, dest='layers',
help='Restrict results to one or more layers (may be repeated)') help='OSM id to lookup in format <NRW><id> (may be repeated)')
_add_api_output_arguments(parser) _add_api_output_arguments(parser)
_add_list_format(parser) _add_list_format(parser)
@@ -296,7 +287,7 @@ class APIReverse:
raise UsageError(ex) from ex raise UsageError(ex) from ex
if result is not None: if result is not None:
locales = _get_locales(args, api.config) locales = _get_locales(args, api.config.DEFAULT_LANGUAGE)
locales.localize_results([result]) locales.localize_results([result])
if args.format == 'debug': if args.format == 'debug':
@@ -361,7 +352,7 @@ class APILookup:
except napi.UsageError as ex: except napi.UsageError as ex:
raise UsageError(ex) from ex raise UsageError(ex) from ex
locales = _get_locales(args, api.config) locales = _get_locales(args, api.config.DEFAULT_LANGUAGE)
locales.localize_results(results) locales.localize_results(results)
if args.format == 'debug': if args.format == 'debug':
@@ -461,7 +452,7 @@ class APIDetails:
raise UsageError(ex) from ex raise UsageError(ex) from ex
if result is not None: if result is not None:
locales = _get_locales(args, api.config) locales = _get_locales(args, api.config.DEFAULT_LANGUAGE)
locales.localize_results([result]) locales.localize_results([result])
if args.format == 'debug': if args.format == 'debug':

View File

@@ -119,7 +119,6 @@ class NominatimArgs:
enable_debug_statements: bool enable_debug_statements: bool
data_object: Sequence[Tuple[str, int]] data_object: Sequence[Tuple[str, int]]
data_area: Sequence[Tuple[str, int]] data_area: Sequence[Tuple[str, int]]
ro_access: bool
# Arguments to 'replication' # Arguments to 'replication'
init: bool init: bool

View File

@@ -154,7 +154,7 @@ async def dump_results(conn: napi.SearchConnection,
await add_result_details(conn, results, await add_result_details(conn, results,
LookupDetails(address_details=True)) LookupDetails(address_details=True))
locale = napi.Locales([lang] if lang else None, conn.config.OUTPUT_NAMES) locale = napi.Locales([lang] if lang else None)
locale.localize_results(results) locale.localize_results(results)
for result in results: for result in results:

View File

@@ -65,8 +65,6 @@ class UpdateRefresh:
help='Update secondary importance raster data') help='Update secondary importance raster data')
group.add_argument('--importance', action='store_true', group.add_argument('--importance', action='store_true',
help='Recompute place importances (expensive!)') help='Recompute place importances (expensive!)')
group.add_argument('--ro-access', action='store_true',
help='Grant read-only access to web user for all tables')
group.add_argument('--website', action='store_true', group.add_argument('--website', action='store_true',
help='DEPRECATED. This function has no function anymore' help='DEPRECATED. This function has no function anymore'
' and will be removed in a future version.') ' and will be removed in a future version.')
@@ -161,11 +159,6 @@ class UpdateRefresh:
LOG.error('WARNING: Website setup is no longer required. ' LOG.error('WARNING: Website setup is no longer required. '
'This function will be removed in future version of Nominatim.') 'This function will be removed in future version of Nominatim.')
if args.ro_access:
from ..tools import admin
LOG.warning('Grant read-only access to web user')
admin.grant_ro_access(args.config.get_libpq_dsn(), args.config)
if args.data_object or args.data_area: if args.data_object or args.data_area:
with connect(args.config.get_libpq_dsn()) as conn: with connect(args.config.get_libpq_dsn()) as conn:
for obj in args.data_object or []: for obj in args.data_object or []:

View File

@@ -23,7 +23,6 @@ from ..tokenizer.base import AbstractTokenizer
from ..version import NOMINATIM_VERSION from ..version import NOMINATIM_VERSION
from .args import NominatimArgs from .args import NominatimArgs
import time
LOG = logging.getLogger() LOG = logging.getLogger()
@@ -87,8 +86,6 @@ class SetupAll:
from ..tools import database_import, postcodes, freeze from ..tools import database_import, postcodes, freeze
from ..indexer.indexer import Indexer from ..indexer.indexer import Indexer
start_time = time.time()
num_threads = args.threads or psutil.cpu_count() or 1 num_threads = args.threads or psutil.cpu_count() or 1
country_info.setup_country_config(args.config) country_info.setup_country_config(args.config)
@@ -141,10 +138,6 @@ class SetupAll:
LOG.warning('Recompute word counts') LOG.warning('Recompute word counts')
tokenizer.update_statistics(args.config, threads=num_threads) tokenizer.update_statistics(args.config, threads=num_threads)
end_time = time.time()
elapsed = end_time - start_time
LOG.warning(f'Import completed successfully in {elapsed:.2f} seconds.')
self._finalize_database(args.config.get_libpq_dsn(), args.offline) self._finalize_database(args.config.get_libpq_dsn(), args.offline)
return 0 return 0

View File

@@ -2,7 +2,7 @@
# #
# This file is part of Nominatim. (https://nominatim.org) # This file is part of Nominatim. (https://nominatim.org)
# #
# Copyright (C) 2026 by the Nominatim developer community. # Copyright (C) 2025 by the Nominatim developer community.
# For a full list of authors see the git log. # For a full list of authors see the git log.
""" """
Nominatim configuration accessor. Nominatim configuration accessor.
@@ -12,7 +12,6 @@ import importlib.util
import logging import logging
import os import os
import sys import sys
import re
from pathlib import Path from pathlib import Path
import json import json
import yaml import yaml
@@ -81,10 +80,6 @@ class Configuration:
self.lib_dir = _LibDirs() self.lib_dir = _LibDirs()
self._private_plugins: Dict[str, object] = {} self._private_plugins: Dict[str, object] = {}
if re.fullmatch(r'[\w-]+', self.DATABASE_WEBUSER) is None:
raise UsageError("Misconfigured DATABASE_WEBUSER. "
"Only alphnumberic characters, - and _ are allowed.")
def set_libdirs(self, **kwargs: StrPath) -> None: def set_libdirs(self, **kwargs: StrPath) -> None:
""" Set paths to library functions and data. """ Set paths to library functions and data.
""" """
@@ -202,7 +197,7 @@ class Configuration:
if dsn.startswith('pgsql:'): if dsn.startswith('pgsql:'):
return dict((p.split('=', 1) for p in dsn[6:].split(';'))) return dict((p.split('=', 1) for p in dsn[6:].split(';')))
return conninfo_to_dict(dsn) # type: ignore return conninfo_to_dict(dsn)
def get_import_style_file(self) -> Path: def get_import_style_file(self) -> Path:
""" Return the import style file as a path object. Translates the """ Return the import style file as a path object. Translates the

View File

@@ -2,7 +2,7 @@
# #
# This file is part of Nominatim. (https://nominatim.org) # This file is part of Nominatim. (https://nominatim.org)
# #
# Copyright (C) 2025 by the Nominatim developer community. # Copyright (C) 2024 by the Nominatim developer community.
# For a full list of authors see the git log. # For a full list of authors see the git log.
""" """
Functions for formatting postcodes according to their country-specific Functions for formatting postcodes according to their country-specific
@@ -29,9 +29,6 @@ class CountryPostcodeMatcher:
self.norm_pattern = re.compile(f'\\s*(?:{country_code.upper()}[ -]?)?({pc_pattern})\\s*') self.norm_pattern = re.compile(f'\\s*(?:{country_code.upper()}[ -]?)?({pc_pattern})\\s*')
self.pattern = re.compile(pc_pattern) self.pattern = re.compile(pc_pattern)
# We want to exclude 0000, 00-000, 000 00 etc
self.zero_pattern = re.compile(r'^[0\- ]+$')
self.output = config.get('output', r'\g<0>') self.output = config.get('output', r'\g<0>')
def match(self, postcode: str) -> Optional[Match[str]]: def match(self, postcode: str) -> Optional[Match[str]]:
@@ -43,10 +40,7 @@ class CountryPostcodeMatcher:
normalized = self.norm_pattern.fullmatch(postcode.upper()) normalized = self.norm_pattern.fullmatch(postcode.upper())
if normalized: if normalized:
match = self.pattern.fullmatch(normalized.group(1)) return self.pattern.fullmatch(normalized.group(1))
if match and self.zero_pattern.match(match.string):
return None
return match
return None return None
@@ -67,15 +61,12 @@ class PostcodeFormatter:
self.country_without_postcode: Set[Optional[str]] = {None} self.country_without_postcode: Set[Optional[str]] = {None}
self.country_matcher = {} self.country_matcher = {}
self.default_matcher = CountryPostcodeMatcher('', {'pattern': '.*'}) self.default_matcher = CountryPostcodeMatcher('', {'pattern': '.*'})
self.postcode_extent: dict[Optional[str], int] = {}
for ccode, prop in country_info.iterate('postcode'): for ccode, prop in country_info.iterate('postcode'):
if prop is False: if prop is False:
self.country_without_postcode.add(ccode) self.country_without_postcode.add(ccode)
elif isinstance(prop, dict): elif isinstance(prop, dict):
self.country_matcher[ccode] = CountryPostcodeMatcher(ccode, prop) self.country_matcher[ccode] = CountryPostcodeMatcher(ccode, prop)
if 'extent' in prop:
self.postcode_extent[ccode] = int(prop['extent'])
else: else:
raise UsageError(f"Invalid entry 'postcode' for country '{ccode}'") raise UsageError(f"Invalid entry 'postcode' for country '{ccode}'")
@@ -116,9 +107,3 @@ class PostcodeFormatter:
`match()` `match()`
""" """
return self.country_matcher.get(country_code, self.default_matcher).normalize(match) return self.country_matcher.get(country_code, self.default_matcher).normalize(match)
def get_postcode_extent(self, country_code: Optional[str]) -> int:
""" Return the extent (in m) to use for the given country. If no
specific extent is set, then the default of 5km will be returned.
"""
return self.postcode_extent.get(country_code, 5000)

View File

@@ -2,7 +2,7 @@
# #
# This file is part of Nominatim. (https://nominatim.org) # This file is part of Nominatim. (https://nominatim.org)
# #
# Copyright (C) 2026 by the Nominatim developer community. # Copyright (C) 2024 by the Nominatim developer community.
# For a full list of authors see the git log. # For a full list of authors see the git log.
""" """
A connection pool that executes incoming queries in parallel. A connection pool that executes incoming queries in parallel.
@@ -27,30 +27,20 @@ class QueryPool:
The results of the queries is discarded. The results of the queries is discarded.
""" """
def __init__(self, dsn: str, pool_size: int = 1, **conn_args: Any) -> None: def __init__(self, dsn: str, pool_size: int = 1, **conn_args: Any) -> None:
self.is_cancelled = False
self.wait_time = 0.0 self.wait_time = 0.0
self.query_queue: 'asyncio.Queue[QueueItem]' = asyncio.Queue(maxsize=2 * pool_size) self.query_queue: 'asyncio.Queue[QueueItem]' = asyncio.Queue(maxsize=2 * pool_size)
self.pool = [asyncio.create_task(self._worker_loop_cancellable(dsn, **conn_args)) self.pool = [asyncio.create_task(self._worker_loop(dsn, **conn_args))
for _ in range(pool_size)] for _ in range(pool_size)]
async def put_query(self, query: psycopg.abc.Query, params: Any) -> None: async def put_query(self, query: psycopg.abc.Query, params: Any) -> None:
""" Schedule a query for execution. """ Schedule a query for execution.
""" """
if self.is_cancelled:
self.clear_queue()
await self.finish()
return
tstart = time.time() tstart = time.time()
await self.query_queue.put((query, params)) await self.query_queue.put((query, params))
self.wait_time += time.time() - tstart self.wait_time += time.time() - tstart
await asyncio.sleep(0) await asyncio.sleep(0)
if self.is_cancelled:
self.clear_queue()
await self.finish()
async def finish(self) -> None: async def finish(self) -> None:
""" Wait for all queries to finish and close the pool. """ Wait for all queries to finish and close the pool.
""" """
@@ -66,25 +56,6 @@ class QueryPool:
if excp is not None: if excp is not None:
raise excp raise excp
def clear_queue(self) -> None:
""" Drop all items silently that might still be queued.
"""
try:
while True:
self.query_queue.get_nowait()
except asyncio.QueueEmpty:
pass # expected
async def _worker_loop_cancellable(self, dsn: str, **conn_args: Any) -> None:
try:
await self._worker_loop(dsn, **conn_args)
except Exception as e:
# Make sure the exception is forwarded to the main function
self.is_cancelled = True
# clear the queue here to ensure that any put() that may be blocked returns
self.clear_queue()
raise e
async def _worker_loop(self, dsn: str, **conn_args: Any) -> None: async def _worker_loop(self, dsn: str, **conn_args: Any) -> None:
conn_args['autocommit'] = True conn_args['autocommit'] = True
aconn = await psycopg.AsyncConnection.connect(dsn, **conn_args) aconn = await psycopg.AsyncConnection.connect(dsn, **conn_args)

View File

@@ -2,13 +2,12 @@
# #
# This file is part of Nominatim. (https://nominatim.org) # This file is part of Nominatim. (https://nominatim.org)
# #
# Copyright (C) 2026 by the Nominatim developer community. # Copyright (C) 2024 by the Nominatim developer community.
# For a full list of authors see the git log. # For a full list of authors see the git log.
""" """
Preprocessing of SQL files. Preprocessing of SQL files.
""" """
from typing import Set, Dict, Any, cast from typing import Set, Dict, Any, cast
import re
import jinja2 import jinja2
@@ -35,9 +34,7 @@ def _get_tables(conn: Connection) -> Set[str]:
with conn.cursor() as cur: with conn.cursor() as cur:
cur.execute("SELECT tablename FROM pg_tables WHERE schemaname = 'public'") cur.execute("SELECT tablename FROM pg_tables WHERE schemaname = 'public'")
# paranoia check: make sure we don't get table names that cause return set((row[0] for row in list(cur)))
# an SQL injection later
return {row[0] for row in list(cur) if re.fullmatch(r'\w+', row[0])}
def _get_middle_db_format(conn: Connection, tables: Set[str]) -> str: def _get_middle_db_format(conn: Connection, tables: Set[str]) -> str:

View File

@@ -2,7 +2,7 @@
# #
# This file is part of Nominatim. (https://nominatim.org) # This file is part of Nominatim. (https://nominatim.org)
# #
# Copyright (C) 2025 by the Nominatim developer community. # Copyright (C) 2024 by the Nominatim developer community.
# For a full list of authors see the git log. # For a full list of authors see the git log.
""" """
Main work horse for indexing (computing addresses) the database. Main work horse for indexing (computing addresses) the database.
@@ -56,10 +56,10 @@ class Indexer:
cur.execute('ANALYZE') cur.execute('ANALYZE')
while True: while True:
if await self.index_by_rank(1, 4) > 0: if await self.index_by_rank(0, 4) > 0:
_analyze() _analyze()
if await self.index_boundaries() > 100: if await self.index_boundaries(0, 30) > 100:
_analyze() _analyze()
if await self.index_by_rank(5, 25) > 100: if await self.index_by_rank(5, 25) > 100:
@@ -68,16 +68,13 @@ class Indexer:
if await self.index_by_rank(26, 30) > 1000: if await self.index_by_rank(26, 30) > 1000:
_analyze() _analyze()
# Special case: rank zero depends on the previously-indexed [1..30] ranks
await self.index_by_rank(0, 0)
if await self.index_postcodes() > 100: if await self.index_postcodes() > 100:
_analyze() _analyze()
if not self.has_pending(): if not self.has_pending():
break break
async def index_boundaries(self, minrank: int = 0, maxrank: int = 30) -> int: async def index_boundaries(self, minrank: int, maxrank: int) -> int:
""" Index only administrative boundaries within the given rank range. """ Index only administrative boundaries within the given rank range.
""" """
total = 0 total = 0
@@ -150,17 +147,14 @@ class Indexer:
total += await self._index(runners.RankRunner(rank, analyzer), total += await self._index(runners.RankRunner(rank, analyzer),
batch=batch, total_tuples=total_tuples.get(rank, 0)) batch=batch, total_tuples=total_tuples.get(rank, 0))
# Special case: rank zero depends on ranks [1..30]
if minrank == 0:
total += await self._index(runners.RankRunner(0, analyzer))
if maxrank == 30: if maxrank == 30:
total += await self._index(runners.RankRunner(0, analyzer))
total += await self._index(runners.InterpolationRunner(analyzer), batch=20) total += await self._index(runners.InterpolationRunner(analyzer), batch=20)
return total return total
async def index_postcodes(self) -> int: async def index_postcodes(self) -> int:
"""Index the entries of the location_postcodes table. """Index the entries of the location_postcode table.
""" """
LOG.warning("Starting indexing postcodes using %s threads", self.num_threads) LOG.warning("Starting indexing postcodes using %s threads", self.num_threads)
@@ -183,7 +177,7 @@ class Indexer:
`total_tuples` may contain the total number of rows to process. `total_tuples` may contain the total number of rows to process.
When not supplied, the value will be computed using the When not supplied, the value will be computed using the
appropriate runner function. approriate runner function.
""" """
LOG.warning("Starting %s (using batch size %s)", runner.name(), batch) LOG.warning("Starting %s (using batch size %s)", runner.name(), batch)

View File

@@ -2,7 +2,7 @@
# #
# This file is part of Nominatim. (https://nominatim.org) # This file is part of Nominatim. (https://nominatim.org)
# #
# Copyright (C) 2025 by the Nominatim developer community. # Copyright (C) 2024 by the Nominatim developer community.
# For a full list of authors see the git log. # For a full list of authors see the git log.
""" """
Mix-ins that provide the actual commands for the indexer for various indexing Mix-ins that provide the actual commands for the indexer for various indexing
@@ -143,22 +143,22 @@ class InterpolationRunner:
class PostcodeRunner(Runner): class PostcodeRunner(Runner):
""" Provides the SQL commands for indexing the location_postcodes table. """ Provides the SQL commands for indexing the location_postcode table.
""" """
def name(self) -> str: def name(self) -> str:
return "postcodes (location_postcodes)" return "postcodes (location_postcode)"
def sql_count_objects(self) -> Query: def sql_count_objects(self) -> Query:
return 'SELECT count(*) FROM location_postcodes WHERE indexed_status > 0' return 'SELECT count(*) FROM location_postcode WHERE indexed_status > 0'
def sql_get_objects(self) -> Query: def sql_get_objects(self) -> Query:
return """SELECT place_id FROM location_postcodes return """SELECT place_id FROM location_postcode
WHERE indexed_status > 0 WHERE indexed_status > 0
ORDER BY country_code, postcode""" ORDER BY country_code, postcode"""
def index_places_query(self, batch_size: int) -> Query: def index_places_query(self, batch_size: int) -> Query:
return pysql.SQL("""UPDATE location_postcodes SET indexed_status = 0 return pysql.SQL("""UPDATE location_postcode SET indexed_status = 0
WHERE place_id IN ({})""")\ WHERE place_id IN ({})""")\
.format(pysql.SQL(',').join((pysql.Placeholder() for _ in range(batch_size)))) .format(pysql.SQL(',').join((pysql.Placeholder() for _ in range(batch_size))))

View File

@@ -71,7 +71,7 @@ class AbstractAnalyzer(ABC):
@abstractmethod @abstractmethod
def update_postcodes_from_db(self) -> None: def update_postcodes_from_db(self) -> None:
""" Update the tokenizer's postcode tokens from the current content """ Update the tokenizer's postcode tokens from the current content
of the `location_postcodes` table. of the `location_postcode` table.
""" """
@abstractmethod @abstractmethod

View File

@@ -2,7 +2,7 @@
# #
# This file is part of Nominatim. (https://nominatim.org) # This file is part of Nominatim. (https://nominatim.org)
# #
# Copyright (C) 2026 by the Nominatim developer community. # Copyright (C) 2025 by the Nominatim developer community.
# For a full list of authors see the git log. # For a full list of authors see the git log.
""" """
Tokenizer implementing normalisation as used before Nominatim 4 but using Tokenizer implementing normalisation as used before Nominatim 4 but using
@@ -294,12 +294,13 @@ class ICUTokenizer(AbstractTokenizer):
with connect(self.dsn) as conn: with connect(self.dsn) as conn:
drop_tables(conn, 'word') drop_tables(conn, 'word')
with conn.cursor() as cur: with conn.cursor() as cur:
cur.execute(pysql.SQL("ALTER TABLE {} RENAME TO word") cur.execute(f"ALTER TABLE {old} RENAME TO word")
.format(pysql.Identifier(old))) for idx in ('word_token', 'word_id'):
for idx in ['word_token', 'word_id'] + [n[0] for n in WORD_TYPES]: cur.execute(f"""ALTER INDEX idx_{old}_{idx}
cur.execute(pysql.SQL("ALTER INDEX {} RENAME TO {}") RENAME TO idx_word_{idx}""")
.format(pysql.Identifier(f"idx_{old}_{idx}"), for name, _ in WORD_TYPES:
pysql.Identifier(f"idx_word_{idx}"))) cur.execute(f"""ALTER INDEX idx_{old}_{name}
RENAME TO idx_word_{name}""")
conn.commit() conn.commit()
@@ -474,23 +475,20 @@ class ICUNameAnalyzer(AbstractAnalyzer):
assert self.conn is not None assert self.conn is not None
word_tokens = set() word_tokens = set()
for name in names: for name in names:
norm_name = self._normalized(name.name) norm_name = self._search_normalized(name.name)
token_name = self._search_normalized(name.name) if norm_name:
if norm_name and token_name: word_tokens.add(norm_name)
word_tokens.add((token_name, norm_name))
with self.conn.cursor() as cur: with self.conn.cursor() as cur:
# Get existing names # Get existing names
cur.execute("""SELECT word_token, cur.execute("""SELECT word_token, coalesce(info ? 'internal', false) as is_internal
word as lookup,
coalesce(info ? 'internal', false) as is_internal
FROM word FROM word
WHERE type = 'C' and info->>'cc' = %s""", WHERE type = 'C' and word = %s""",
(country_code, )) (country_code, ))
# internal/external names # internal/external names
existing_tokens: Dict[bool, Set[Tuple[str, str]]] = {True: set(), False: set()} existing_tokens: Dict[bool, Set[str]] = {True: set(), False: set()}
for word in cur: for word in cur:
existing_tokens[word[2]].add((word[0], word[1])) existing_tokens[word[1]].add(word[0])
# Delete names that no longer exist. # Delete names that no longer exist.
gone_tokens = existing_tokens[internal] - word_tokens gone_tokens = existing_tokens[internal] - word_tokens
@@ -498,10 +496,10 @@ class ICUNameAnalyzer(AbstractAnalyzer):
gone_tokens.update(existing_tokens[False] & word_tokens) gone_tokens.update(existing_tokens[False] & word_tokens)
if gone_tokens: if gone_tokens:
cur.execute("""DELETE FROM word cur.execute("""DELETE FROM word
USING jsonb_array_elements(%s) as data USING unnest(%s::text[]) as token
WHERE type = 'C' and info->>'cc' = %s WHERE type = 'C' and word = %s
and word_token = data->>0 and word = data->>1""", and word_token = token""",
(Jsonb(list(gone_tokens)), country_code)) (list(gone_tokens), country_code))
# Only add those names that are not yet in the list. # Only add those names that are not yet in the list.
new_tokens = word_tokens - existing_tokens[True] new_tokens = word_tokens - existing_tokens[True]
@@ -510,17 +508,15 @@ class ICUNameAnalyzer(AbstractAnalyzer):
if new_tokens: if new_tokens:
if internal: if internal:
sql = """INSERT INTO word (word_token, type, word, info) sql = """INSERT INTO word (word_token, type, word, info)
(SELECT data->>0, 'C', data->>1, (SELECT token, 'C', %s, '{"internal": "yes"}'
jsonb_build_object('internal', 'yes', 'cc', %s::text) FROM unnest(%s::text[]) as token)
FROM jsonb_array_elements(%s) as data)
""" """
else: else:
sql = """INSERT INTO word (word_token, type, word, info) sql = """INSERT INTO word (word_token, type, word)
(SELECT data->>0, 'C', data->>1, (SELECT token, 'C', %s
jsonb_build_object('cc', %s::text) FROM unnest(%s::text[]) as token)
FROM jsonb_array_elements(%s) as data)
""" """
cur.execute(sql, (country_code, Jsonb(list(new_tokens)))) cur.execute(sql, (country_code, list(new_tokens)))
def process_place(self, place: PlaceInfo) -> Mapping[str, Any]: def process_place(self, place: PlaceInfo) -> Mapping[str, Any]:
""" Determine tokenizer information about the given place. """ Determine tokenizer information about the given place.

View File

@@ -16,7 +16,6 @@ from psycopg.types.json import Json
from ..typing import DictCursorResult from ..typing import DictCursorResult
from ..config import Configuration from ..config import Configuration
from ..db.connection import connect, Cursor, register_hstore from ..db.connection import connect, Cursor, register_hstore
from ..db.sql_preprocessor import SQLPreprocessor
from ..errors import UsageError from ..errors import UsageError
from ..tokenizer import factory as tokenizer_factory from ..tokenizer import factory as tokenizer_factory
from ..data.place_info import PlaceInfo from ..data.place_info import PlaceInfo
@@ -106,12 +105,3 @@ def clean_deleted_relations(config: Configuration, age: str) -> None:
except psycopg.DataError as exc: except psycopg.DataError as exc:
raise UsageError('Invalid PostgreSQL time interval format') from exc raise UsageError('Invalid PostgreSQL time interval format') from exc
conn.commit() conn.commit()
def grant_ro_access(dsn: str, config: Configuration) -> None:
""" Grant read-only access to the web user for all Nominatim tables.
This can be used to grant access to a different user after import.
"""
with connect(dsn) as conn:
sql = SQLPreprocessor(conn, config)
sql.run_sql_file(conn, 'grants.sql')

View File

@@ -113,8 +113,8 @@ def _get_indexes(conn: Connection) -> List[str]:
'idx_placex_geometry_placenode', 'idx_placex_geometry_placenode',
'idx_osmline_parent_place_id', 'idx_osmline_parent_place_id',
'idx_osmline_parent_osm_id', 'idx_osmline_parent_osm_id',
'idx_location_postcodes_id', 'idx_postcode_id',
'idx_location_postcodes_postcode' 'idx_postcode_postcode'
] ]
# These won't exist if --reverse-only import was used # These won't exist if --reverse-only import was used

View File

@@ -2,7 +2,7 @@
# #
# This file is part of Nominatim. (https://nominatim.org) # This file is part of Nominatim. (https://nominatim.org)
# #
# Copyright (C) 2025 by the Nominatim developer community. # Copyright (C) 2024 by the Nominatim developer community.
# For a full list of authors see the git log. # For a full list of authors see the git log.
""" """
Functions for setting up and importing a new Nominatim database. Functions for setting up and importing a new Nominatim database.
@@ -152,11 +152,10 @@ def create_tables(conn: Connection, config: Configuration, reverse_only: bool =
When `reverse_only` is True, then the main table for searching will When `reverse_only` is True, then the main table for searching will
be skipped and only reverse search is possible. be skipped and only reverse search is possible.
""" """
SQLPreprocessor(conn, config).run_sql_file(conn, 'tables.sql', sql = SQLPreprocessor(conn, config)
create_reverse_only=reverse_only) sql.env.globals['db']['reverse_only'] = reverse_only
# reinitiate the preprocessor to get all the newly created tables sql.run_sql_file(conn, 'tables.sql')
SQLPreprocessor(conn, config).run_sql_file(conn, 'grants.sql')
def create_table_triggers(conn: Connection, config: Configuration) -> None: def create_table_triggers(conn: Connection, config: Configuration) -> None:
@@ -184,7 +183,7 @@ def truncate_data_tables(conn: Connection) -> None:
cur.execute('TRUNCATE location_area_country') cur.execute('TRUNCATE location_area_country')
cur.execute('TRUNCATE location_property_tiger') cur.execute('TRUNCATE location_property_tiger')
cur.execute('TRUNCATE location_property_osmline') cur.execute('TRUNCATE location_property_osmline')
cur.execute('TRUNCATE location_postcodes') cur.execute('TRUNCATE location_postcode')
if table_exists(conn, 'search_name'): if table_exists(conn, 'search_name'):
cur.execute('TRUNCATE search_name') cur.execute('TRUNCATE search_name')
cur.execute('DROP SEQUENCE IF EXISTS seq_place') cur.execute('DROP SEQUENCE IF EXISTS seq_place')
@@ -194,7 +193,7 @@ def truncate_data_tables(conn: Connection) -> None:
WHERE tablename LIKE 'location_road_%'""") WHERE tablename LIKE 'location_road_%'""")
for table in [r[0] for r in list(cur)]: for table in [r[0] for r in list(cur)]:
cur.execute(pysql.SQL('TRUNCATE {}').format(pysql.Identifier(table))) cur.execute('TRUNCATE ' + table)
conn.commit() conn.commit()
@@ -226,7 +225,7 @@ async def load_data(dsn: str, threads: int) -> None:
total=pysql.Literal(placex_threads), total=pysql.Literal(placex_threads),
mod=pysql.Literal(imod)), None) mod=pysql.Literal(imod)), None)
# Interpolations need to be copied separately # Interpolations need to be copied seperately
await pool.put_query(""" await pool.put_query("""
INSERT INTO location_property_osmline (osm_id, address, linegeo) INSERT INTO location_property_osmline (osm_id, address, linegeo)
SELECT osm_id, address, geometry FROM place SELECT osm_id, address, geometry FROM place

View File

@@ -2,7 +2,7 @@
# #
# This file is part of Nominatim. (https://nominatim.org) # This file is part of Nominatim. (https://nominatim.org)
# #
# Copyright (C) 2025 by the Nominatim developer community. # Copyright (C) 2024 by the Nominatim developer community.
# For a full list of authors see the git log. # For a full list of authors see the git log.
""" """
Functions for removing unnecessary data from the database. Functions for removing unnecessary data from the database.
@@ -18,11 +18,10 @@ UPDATE_TABLES = [
'address_levels', 'address_levels',
'gb_postcode', 'gb_postcode',
'import_osmosis_log', 'import_osmosis_log',
'import_polygon_%',
'location_area%', 'location_area%',
'location_road%', 'location_road%',
'place', 'place',
'place_entrance',
'place_postcode',
'planet_osm_%', 'planet_osm_%',
'search_name_%', 'search_name_%',
'us_postcode', 'us_postcode',

View File

@@ -2,7 +2,7 @@
# #
# This file is part of Nominatim. (https://nominatim.org) # This file is part of Nominatim. (https://nominatim.org)
# #
# Copyright (C) 2025 by the Nominatim developer community. # Copyright (C) 2024 by the Nominatim developer community.
# For a full list of authors see the git log. # For a full list of authors see the git log.
""" """
Functions for database migration to newer software versions. Functions for database migration to newer software versions.
@@ -18,8 +18,6 @@ from ..db.connection import connect, Connection, \
from ..db.sql_preprocessor import SQLPreprocessor from ..db.sql_preprocessor import SQLPreprocessor
from ..version import NominatimVersion, NOMINATIM_VERSION, parse_version from ..version import NominatimVersion, NOMINATIM_VERSION, parse_version
from ..tokenizer import factory as tokenizer_factory from ..tokenizer import factory as tokenizer_factory
from ..data.country_info import create_country_names, setup_country_config
from .freeze import is_frozen
from . import refresh from . import refresh
LOG = logging.getLogger() LOG = logging.getLogger()
@@ -29,7 +27,7 @@ _MIGRATION_FUNCTIONS: List[Tuple[NominatimVersion, Callable[..., None]]] = []
def migrate(config: Configuration, paths: Any) -> int: def migrate(config: Configuration, paths: Any) -> int:
""" Check for the current database version and execute migrations, """ Check for the current database version and execute migrations,
if necessary. if necesssary.
""" """
with connect(config.get_libpq_dsn()) as conn: with connect(config.get_libpq_dsn()) as conn:
register_hstore(conn) register_hstore(conn)
@@ -143,7 +141,7 @@ def create_placex_entrance_table(conn: Connection, config: Configuration, **_: A
@_migration(5, 1, 99, 1) @_migration(5, 1, 99, 1)
def create_place_entrance_table(conn: Connection, config: Configuration, **_: Any) -> None: def create_place_entrance_table(conn: Connection, config: Configuration, **_: Any) -> None:
""" Add the place_entrance table to store incoming entrance nodes """ Add the place_entrance table to store incomming entrance nodes
""" """
if not table_exists(conn, 'place_entrance'): if not table_exists(conn, 'place_entrance'):
with conn.cursor() as cur: with conn.cursor() as cur:
@@ -158,195 +156,3 @@ def create_place_entrance_table(conn: Connection, config: Configuration, **_: An
CREATE UNIQUE INDEX place_entrance_osm_id_idx ON place_entrance CREATE UNIQUE INDEX place_entrance_osm_id_idx ON place_entrance
USING BTREE (osm_id); USING BTREE (osm_id);
""") """)
@_migration(5, 2, 99, 0)
def convert_country_tokens(conn: Connection, config: Configuration, **_: Any) -> None:
""" Convert country word tokens
Country tokens now save the country in the info field instead of the
word. This migration removes all country tokens from the word table
and reimports the default country name. This means that custom names
are lost. If you need them back, invalidate the OSM objects containing
the names by setting indexed_status to 2 and then reindex the database.
"""
tokenizer = tokenizer_factory.get_tokenizer_for_db(config)
# There is only one tokenizer at the time of migration, so we make
# some assumptions here about the structure of the database. This will
# fail if somebody has written a custom tokenizer.
with conn.cursor() as cur:
cur.execute("DELETE FROM word WHERE type = 'C'")
conn.commit()
setup_country_config(config)
create_country_names(conn, tokenizer, config.get_str_list('LANGUAGES'))
@_migration(5, 2, 99, 1)
def create_place_postcode_table(conn: Connection, config: Configuration, **_: Any) -> None:
""" Restructure postcode tables
"""
sqlp = SQLPreprocessor(conn, config)
mutable = not is_frozen(conn)
has_place_table = table_exists(conn, 'place_postcode')
has_postcode_table = table_exists(conn, 'location_postcodes')
if mutable and not has_place_table:
with conn.cursor() as cur:
cur.execute(
"""
CREATE TABLE place_postcode (
osm_type CHAR(1) NOT NULL,
osm_id BIGINT NOT NULL,
postcode TEXT NOT NULL,
country_code TEXT,
centroid GEOMETRY(Point, 4326) NOT NULL,
geometry GEOMETRY(Geometry, 4326)
)
""")
# Move postcode points into the new table
cur.execute("ALTER TABLE place DISABLE TRIGGER ALL")
cur.execute(
"""
WITH deleted AS (
DELETE FROM place
WHERE (class = 'place' AND type = 'postcode')
OR (osm_type = 'R'
AND class = 'boundary' AND type = 'postal_code')
RETURNING osm_type, osm_id, address->'postcode' as postcode,
ST_Centroid(geometry) as centroid,
(CASE WHEN class = 'place' THEN NULL ELSE geometry END) as geometry)
INSERT INTO place_postcode (osm_type, osm_id, postcode, centroid, geometry)
(SELECT * FROM deleted
WHERE deleted.postcode is not NULL AND deleted.centroid is not NULL)
""")
cur.execute(
"""
CREATE INDEX place_postcode_osm_id_idx ON place_postcode
USING BTREE (osm_type, osm_id)
""")
cur.execute(
"""
CREATE INDEX place_postcode_postcode_idx ON place_postcode
USING BTREE (postcode)
""")
cur.execute("ALTER TABLE place ENABLE TRIGGER ALL")
if not has_postcode_table:
sqlp.run_sql_file(conn, 'functions/postcode_triggers.sql')
with conn.cursor() as cur:
# create a new location_postcode table which will replace the
# old one atomically in the end
cur.execute(
"""
CREATE TABLE location_postcodes (
place_id BIGINT,
osm_id BIGINT,
rank_search SMALLINT,
parent_place_id BIGINT,
indexed_status SMALLINT,
indexed_date TIMESTAMP,
country_code VARCHAR(2),
postcode TEXT,
centroid Geometry(Point, 4326),
geometry Geometry(Geometry, 4326) NOT NULL
)
""")
sqlp.run_string(conn,
'GRANT SELECT ON location_postcodes TO "{{config.DATABASE_WEBUSER}}"')
# remove postcodes from the various auxiliary tables
cur.execute(
"""
DELETE FROM place_addressline
WHERE address_place_id = ANY(
SELECT place_id FROM placex
WHERE osm_type = 'R'
AND class = 'boundary' AND type = 'postal_code')
""")
if mutable:
cur.execute(
"""
SELECT deleteLocationArea(partition, place_id, rank_search),
deleteSearchName(partition, place_id)
FROM placex
WHERE osm_type = 'R' AND class = 'boundary' AND type = 'postal_code'
""")
if table_exists(conn, 'search_name'):
cur.execute(
"""
DELETE FROM search_name
WHERE place_id = ANY(
SELECT place_id FROM placex
WHERE osm_type = 'R'
AND class = 'boundary' AND type = 'postal_code')
""")
# move postcode areas from placex to location_postcodes
# avoiding automatic invalidation
cur.execute("ALTER TABLE placex DISABLE TRIGGER ALL")
cur.execute(
"""
WITH deleted AS (
DELETE FROM placex
WHERE osm_type = 'R'
AND class = 'boundary' AND type = 'postal_code'
RETURNING place_id, osm_id, rank_search, parent_place_id,
indexed_status, indexed_date,
country_code, postcode, centroid, geometry)
INSERT INTO location_postcodes (SELECT * from deleted)
""")
cur.execute("ALTER TABLE placex ENABLE TRIGGER ALL")
# remove any old postcode centroid that would overlap with areas
cur.execute(
"""
DELETE FROM location_postcode o USING location_postcodes n
WHERE o.country_code = n.country_code
AND o.postcode = n.postcode
""")
# copy over old postcodes
cur.execute(
"""
INSERT INTO location_postcodes
(SELECT place_id, NULL, rank_search, parent_place_id,
indexed_status, indexed_date, country_code,
postcode, geometry,
ST_Expand(geometry, 0.05)
FROM location_postcode)
""")
# add indexes and triggers
cur.execute("""CREATE INDEX idx_location_postcodes_geometry
ON location_postcodes USING GIST(geometry)""")
cur.execute("""CREATE INDEX idx_location_postcodes_id
ON location_postcodes USING BTREE(place_id)""")
cur.execute("""CREATE INDEX idx_location_postcodes_osmid
ON location_postcodes USING BTREE(osm_id)""")
cur.execute("""CREATE INDEX idx_location_postcodes_postcode
ON location_postcodes USING BTREE(postcode, country_code)""")
cur.execute("""CREATE INDEX idx_location_postcodes_parent_place_id
ON location_postcodes USING BTREE(parent_place_id)""")
cur.execute("""CREATE TRIGGER location_postcodes_before_update
BEFORE UPDATE ON location_postcodes
FOR EACH ROW EXECUTE PROCEDURE postcodes_update()""")
cur.execute("""CREATE TRIGGER location_postcodes_before_delete
BEFORE DELETE ON location_postcodes
FOR EACH ROW EXECUTE PROCEDURE postcodes_delete()""")
cur.execute("""CREATE TRIGGER location_postcodes_before_insert
BEFORE INSERT ON location_postcodes
FOR EACH ROW EXECUTE PROCEDURE postcodes_insert()""")
sqlp.run_string(
conn,
"""
CREATE INDEX IF NOT EXISTS idx_placex_geometry_reverse_lookupPolygon_nopostcode
ON placex USING gist (geometry) {{db.tablespace.search_index}}
WHERE St_GeometryType(geometry) in ('ST_Polygon', 'ST_MultiPolygon')
AND rank_address between 4 and 25
AND name is not null AND linked_place_id is null;
CREATE INDEX IF NOT EXISTS idx_placex_geometry_reverse_lookupPlaceNode_nopostcode
ON placex USING gist (ST_Buffer(geometry, reverse_place_diameter(rank_search)))
{{db.tablespace.search_index}}
WHERE rank_address between 4 and 25
AND name is not null AND linked_place_id is null AND osm_type = 'N';
CREATE INDEX idx_placex_geometry_placenode_nopostcode ON placex
USING SPGIST (geometry) {{db.tablespace.address_index}}
WHERE osm_type = 'N' and rank_search < 26 and class = 'place';
ANALYSE;
""")

View File

@@ -8,7 +8,7 @@
Functions for importing, updating and otherwise maintaining the table Functions for importing, updating and otherwise maintaining the table
of artificial postcode centroids. of artificial postcode centroids.
""" """
from typing import Optional, Tuple, Dict, TextIO from typing import Optional, Tuple, Dict, List, TextIO
from collections import defaultdict from collections import defaultdict
from pathlib import Path from pathlib import Path
import csv import csv
@@ -38,26 +38,13 @@ def _to_float(numstr: str, max_value: float) -> float:
return num return num
def _extent_to_rank(extent: int) -> int:
""" Guess a suitable search rank from the extent of a postcode.
"""
if extent <= 100:
return 25
if extent <= 3000:
return 23
return 21
class _PostcodeCollector: class _PostcodeCollector:
""" Collector for postcodes of a single country. """ Collector for postcodes of a single country.
""" """
def __init__(self, country: str, matcher: Optional[CountryPostcodeMatcher], def __init__(self, country: str, matcher: Optional[CountryPostcodeMatcher]):
default_extent: int, exclude: set[str] = set()):
self.country = country self.country = country
self.matcher = matcher self.matcher = matcher
self.extent = default_extent
self.exclude = exclude
self.collected: Dict[str, PointsCentroid] = defaultdict(PointsCentroid) self.collected: Dict[str, PointsCentroid] = defaultdict(PointsCentroid)
self.normalization_cache: Optional[Tuple[str, Optional[str]]] = None self.normalization_cache: Optional[Tuple[str, Optional[str]]] = None
@@ -74,7 +61,7 @@ class _PostcodeCollector:
normalized = self.matcher.normalize(match) if match else None normalized = self.matcher.normalize(match) if match else None
self.normalization_cache = (postcode, normalized) self.normalization_cache = (postcode, normalized)
if normalized and normalized not in self.exclude: if normalized:
self.collected[normalized] += (x, y) self.collected[normalized] += (x, y)
def commit(self, conn: Connection, analyzer: AbstractAnalyzer, def commit(self, conn: Connection, analyzer: AbstractAnalyzer,
@@ -86,38 +73,61 @@ class _PostcodeCollector:
""" """
if project_dir is not None: if project_dir is not None:
self._update_from_external(analyzer, project_dir) self._update_from_external(analyzer, project_dir)
to_add, to_delete, to_update = self._compute_changes(conn)
with conn.cursor() as cur: LOG.info("Processing country '%s' (%s added, %s deleted, %s updated).",
cur.execute("""SELECT postcode FROM location_postcodes self.country, len(to_add), len(to_delete), len(to_update))
WHERE country_code = %s AND osm_id is null""",
(self.country, ))
to_delete = [row[0] for row in cur if row[0] not in self.collected]
to_add = [dict(zip(('pc', 'x', 'y'), (k, *v.centroid())))
for k, v in self.collected.items()]
self.collected = defaultdict(PointsCentroid)
LOG.info("Processing country '%s' (%s added, %s deleted).",
self.country, len(to_add), len(to_delete))
with conn.cursor() as cur: with conn.cursor() as cur:
if to_add: if to_add:
cur.executemany(pysql.SQL( cur.executemany(pysql.SQL(
"""INSERT INTO location_postcodes """INSERT INTO location_postcode
(country_code, rank_search, postcode, centroid, geometry) (place_id, indexed_status, country_code,
VALUES ({}, {}, %(pc)s, postcode, geometry)
ST_SetSRID(ST_MakePoint(%(x)s, %(y)s), 4326), VALUES (nextval('seq_place'), 1, {}, %s,
expand_by_meters(ST_SetSRID(ST_MakePoint(%(x)s, %(y)s), 4326), {})) ST_SetSRID(ST_MakePoint(%s, %s), 4326))
""").format(pysql.Literal(self.country), """).format(pysql.Literal(self.country)),
pysql.Literal(_extent_to_rank(self.extent)),
pysql.Literal(self.extent)),
to_add) to_add)
if to_delete: if to_delete:
cur.execute("""DELETE FROM location_postcodes cur.execute("""DELETE FROM location_postcode
WHERE country_code = %s and postcode = any(%s) WHERE country_code = %s and postcode = any(%s)
AND osm_id is null
""", (self.country, to_delete)) """, (self.country, to_delete))
cur.execute("ANALYSE location_postcodes") if to_update:
cur.executemany(
pysql.SQL("""UPDATE location_postcode
SET indexed_status = 2,
geometry = ST_SetSRID(ST_Point(%s, %s), 4326)
WHERE country_code = {} and postcode = %s
""").format(pysql.Literal(self.country)),
to_update)
def _compute_changes(
self, conn: Connection
) -> Tuple[List[Tuple[str, float, float]], List[str], List[Tuple[float, float, str]]]:
""" Compute which postcodes from the collected postcodes have to be
added or modified and which from the location_postcode table
have to be deleted.
"""
to_update = []
to_delete = []
with conn.cursor() as cur:
cur.execute("""SELECT postcode, ST_X(geometry), ST_Y(geometry)
FROM location_postcode
WHERE country_code = %s""",
(self.country, ))
for postcode, x, y in cur:
pcobj = self.collected.pop(postcode, None)
if pcobj:
newx, newy = pcobj.centroid()
if abs(x - newx) > 0.0000001 or abs(y - newy) > 0.0000001:
to_update.append((newx, newy, postcode))
else:
to_delete.append(postcode)
to_add = [(k, *v.centroid()) for k, v in self.collected.items()]
self.collected = defaultdict(PointsCentroid)
return to_add, to_delete, to_update
def _update_from_external(self, analyzer: AbstractAnalyzer, project_dir: Path) -> None: def _update_from_external(self, analyzer: AbstractAnalyzer, project_dir: Path) -> None:
""" Look for an external postcode file for the active country in """ Look for an external postcode file for the active country in
@@ -142,7 +152,7 @@ class _PostcodeCollector:
_to_float(row['lat'], 90)) _to_float(row['lat'], 90))
self.collected[postcode] += centroid self.collected[postcode] += centroid
except ValueError: except ValueError:
LOG.warning("Bad coordinates %s, %s in '%s' country postcode file.", LOG.warning("Bad coordinates %s, %s in %s country postcode file.",
row['lat'], row['lon'], self.country) row['lat'], row['lon'], self.country)
finally: finally:
@@ -159,137 +169,41 @@ class _PostcodeCollector:
if fname.is_file(): if fname.is_file():
LOG.info("Using external postcode file '%s'.", fname) LOG.info("Using external postcode file '%s'.", fname)
return gzip.open(fname, 'rt', encoding='utf-8') return gzip.open(fname, 'rt')
return None return None
def update_postcodes(dsn: str, project_dir: Optional[Path], tokenizer: AbstractTokenizer) -> None: def update_postcodes(dsn: str, project_dir: Optional[Path], tokenizer: AbstractTokenizer) -> None:
""" Update the table of postcodes from the input tables """ Update the table of artificial postcodes.
placex and place_postcode.
Computes artificial postcode centroids from the placex table,
potentially enhances it with external data and then updates the
postcodes in the table 'location_postcode'.
""" """
matcher = PostcodeFormatter() matcher = PostcodeFormatter()
with tokenizer.name_analyzer() as analyzer: with tokenizer.name_analyzer() as analyzer:
with connect(dsn) as conn: with connect(dsn) as conn:
# Backfill country_code column where required
conn.execute("""UPDATE place_postcode
SET country_code = get_country_code(centroid)
WHERE country_code is null
""")
# Now update first postcode areas
_update_postcode_areas(conn, analyzer, matcher)
# Then fill with estimated postcode centroids from other info
_update_guessed_postcode(conn, analyzer, matcher, project_dir)
conn.commit()
analyzer.update_postcodes_from_db()
def _insert_postcode_areas(conn: Connection, country_code: str,
extent: int, pcs: list[dict[str, str]]) -> None:
if pcs:
with conn.cursor() as cur:
cur.executemany(
pysql.SQL(
""" INSERT INTO location_postcodes
(osm_id, country_code, rank_search, postcode, centroid, geometry)
SELECT osm_id, country_code, {}, %(out)s, centroid, geometry
FROM place_postcode
WHERE osm_type = 'R'
and country_code = {} and postcode = %(in)s
and geometry is not null
""").format(pysql.Literal(_extent_to_rank(extent)),
pysql.Literal(country_code)),
pcs)
def _update_postcode_areas(conn: Connection, analyzer: AbstractAnalyzer,
matcher: PostcodeFormatter) -> None:
""" Update the postcode areas made from postcode boundaries.
"""
# first delete all areas that have gone
conn.execute(""" DELETE FROM location_postcodes pc
WHERE pc.osm_id is not null
AND NOT EXISTS(
SELECT * FROM place_postcode pp
WHERE pp.osm_type = 'R' and pp.osm_id = pc.osm_id
and geometry is not null)
""")
# now insert all in country batches, triggers will ensure proper updates
with conn.cursor() as cur:
cur.execute(""" SELECT country_code, postcode FROM place_postcode
WHERE geometry is not null and osm_type = 'R'
ORDER BY country_code
""")
country_code = None
fmt = None
pcs = []
for cc, postcode in cur:
if country_code is None:
country_code = cc
fmt = matcher.get_matcher(country_code)
elif country_code != cc:
_insert_postcode_areas(conn, country_code,
matcher.get_postcode_extent(country_code), pcs)
country_code = cc
fmt = matcher.get_matcher(country_code)
pcs = []
if fmt is not None:
if (m := fmt.match(postcode)):
pcs.append({'out': fmt.normalize(m), 'in': postcode})
if country_code is not None and pcs:
_insert_postcode_areas(conn, country_code,
matcher.get_postcode_extent(country_code), pcs)
def _update_guessed_postcode(conn: Connection, analyzer: AbstractAnalyzer,
matcher: PostcodeFormatter, project_dir: Optional[Path]) -> None:
""" Computes artificial postcode centroids from the placex table,
potentially enhances it with external data and then updates the
postcodes in the table 'location_postcodes'.
"""
# First get the list of countries that currently have postcodes. # First get the list of countries that currently have postcodes.
# (Doing this before starting to insert, so it is fast on import.) # (Doing this before starting to insert, so it is fast on import.)
with conn.cursor() as cur: with conn.cursor() as cur:
cur.execute("""SELECT DISTINCT country_code FROM location_postcodes cur.execute("SELECT DISTINCT country_code FROM location_postcode")
WHERE osm_id is null""") todo_countries = set((row[0] for row in cur))
todo_countries = {row[0] for row in cur}
# Next, get the list of postcodes that are already covered by areas.
area_pcs = defaultdict(set)
with conn.cursor() as cur:
cur.execute("""SELECT country_code, postcode
FROM location_postcodes WHERE osm_id is not null
ORDER BY country_code""")
for cc, pc in cur:
area_pcs[cc].add(pc)
# Create a temporary table which contains coverage of the postcode areas.
with conn.cursor() as cur:
cur.execute("DROP TABLE IF EXISTS _global_postcode_area")
cur.execute("""CREATE TABLE _global_postcode_area AS
(SELECT ST_SubDivide(ST_SimplifyPreserveTopology(
ST_Union(geometry), 0.00001), 128) as geometry
FROM place_postcode WHERE geometry is not null)
""")
cur.execute("CREATE INDEX ON _global_postcode_area USING gist(geometry)")
# Recompute the list of valid postcodes from placex. # Recompute the list of valid postcodes from placex.
with conn.cursor(name="placex_postcodes") as cur: with conn.cursor(name="placex_postcodes") as cur:
cur.execute(""" cur.execute("""
SELECT country_code, postcode, ST_X(centroid), ST_Y(centroid) SELECT cc, pc, ST_X(centroid), ST_Y(centroid)
FROM ( FROM (SELECT
(SELECT country_code, address->'postcode' as postcode, centroid COALESCE(plx.country_code,
FROM placex WHERE address ? 'postcode') get_country_code(ST_Centroid(pl.geometry))) as cc,
UNION pl.address->'postcode' as pc,
(SELECT country_code, postcode, centroid COALESCE(plx.centroid, ST_Centroid(pl.geometry)) as centroid
FROM place_postcode WHERE geometry is null) FROM place AS pl LEFT OUTER JOIN placex AS plx
) x ON pl.osm_id = plx.osm_id AND pl.osm_type = plx.osm_type
WHERE not postcode like '%,%' and not postcode like '%;%' WHERE pl.address ? 'postcode' AND pl.geometry IS NOT null) xx
AND NOT EXISTS(SELECT * FROM _global_postcode_area g WHERE pc IS NOT null AND cc IS NOT null
WHERE ST_Intersects(x.centroid, g.geometry)) ORDER BY cc, pc""")
ORDER BY country_code""")
collector = None collector = None
@@ -297,9 +211,7 @@ def _update_guessed_postcode(conn: Connection, analyzer: AbstractAnalyzer,
if collector is None or country != collector.country: if collector is None or country != collector.country:
if collector is not None: if collector is not None:
collector.commit(conn, analyzer, project_dir) collector.commit(conn, analyzer, project_dir)
collector = _PostcodeCollector(country, matcher.get_matcher(country), collector = _PostcodeCollector(country, matcher.get_matcher(country))
matcher.get_postcode_extent(country),
exclude=area_pcs[country])
todo_countries.discard(country) todo_countries.discard(country)
collector.add(postcode, x, y) collector.add(postcode, x, y)
@@ -309,15 +221,17 @@ def _update_guessed_postcode(conn: Connection, analyzer: AbstractAnalyzer,
# Now handle any countries that are only in the postcode table. # Now handle any countries that are only in the postcode table.
for country in todo_countries: for country in todo_countries:
fmt = matcher.get_matcher(country) fmt = matcher.get_matcher(country)
ext = matcher.get_postcode_extent(country) _PostcodeCollector(country, fmt).commit(conn, analyzer, project_dir)
_PostcodeCollector(country, fmt, ext,
exclude=area_pcs[country]).commit(conn, analyzer, project_dir)
conn.execute("DROP TABLE IF EXISTS _global_postcode_area") conn.commit()
analyzer.update_postcodes_from_db()
def can_compute(dsn: str) -> bool: def can_compute(dsn: str) -> bool:
""" Check that the necessary tables exist so that postcodes can be computed. """
Check that the place table exists so that
postcodes can be computed.
""" """
with connect(dsn) as conn: with connect(dsn) as conn:
return table_exists(conn, 'place_postcode') return table_exists(conn, 'place')

View File

@@ -141,9 +141,7 @@ def import_importance_csv(dsn: str, data_file: Path) -> int:
copy_cmd = """COPY wikimedia_importance(language, title, importance, wikidata) copy_cmd = """COPY wikimedia_importance(language, title, importance, wikidata)
FROM STDIN""" FROM STDIN"""
with gzip.open( with gzip.open(str(data_file), 'rt') as fd, cur.copy(copy_cmd) as copy:
str(data_file), 'rt', encoding='utf-8') as fd, \
cur.copy(copy_cmd) as copy:
for row in csv.DictReader(fd, delimiter='\t', quotechar='|'): for row in csv.DictReader(fd, delimiter='\t', quotechar='|'):
wd_id = int(row['wikidata_id'][1:]) wd_id = int(row['wikidata_id'][1:])
copy.write_row((row['language'], copy.write_row((row['language'],

View File

@@ -11,8 +11,6 @@ from typing import Iterable
import re import re
import logging import logging
import mwparserfromhell
from ...config import Configuration from ...config import Configuration
from ...utils.url_utils import get_url from ...utils.url_utils import get_url
from .special_phrase import SpecialPhrase from .special_phrase import SpecialPhrase
@@ -38,6 +36,10 @@ class SPWikiLoader:
""" """
def __init__(self, config: Configuration) -> None: def __init__(self, config: Configuration) -> None:
self.config = config self.config = config
# Compile the regex here to increase performances.
self.occurence_pattern = re.compile(
r'\| *([^\|]+) *\|\| *([^\|]+) *\|\| *([^\|]+) *\|\| *([^\|]+) *\|\| *([\-YN])'
)
# Hack around a bug where building=yes was imported with quotes into the wiki # Hack around a bug where building=yes was imported with quotes into the wiki
self.type_fix_pattern = re.compile(r'\"|&quot;') self.type_fix_pattern = re.compile(r'\"|&quot;')
@@ -56,21 +58,11 @@ class SPWikiLoader:
LOG.warning('Importing phrases for lang: %s...', lang) LOG.warning('Importing phrases for lang: %s...', lang)
loaded_xml = _get_wiki_content(lang) loaded_xml = _get_wiki_content(lang)
wikicode = mwparserfromhell.parse(loaded_xml) # One match will be of format [label, class, type, operator, plural]
matches = self.occurence_pattern.findall(loaded_xml)
for table in wikicode.filter_tags(matches=lambda t: t.tag == 'table'): for match in matches:
for row in table.contents.filter_tags(matches=lambda t: t.tag == 'tr'): yield SpecialPhrase(match[0],
cells = list(row.contents.filter_tags(matches=lambda t: t.tag == 'td')) match[1],
self.type_fix_pattern.sub('', match[2]),
if len(cells) < 5: match[3])
continue
label = cells[0].contents.strip_code().strip()
cls = cells[1].contents.strip_code().strip()
typ = cells[2].contents.strip_code().strip()
operator = cells[3].contents.strip_code().strip()
yield SpecialPhrase(label,
cls,
self.type_fix_pattern.sub('', typ),
operator)

View File

@@ -17,12 +17,13 @@ import tarfile
from psycopg.types.json import Json from psycopg.types.json import Json
from ..config import Configuration from ..config import Configuration
from ..db.connection import connect, table_exists from ..db.connection import connect
from ..db.sql_preprocessor import SQLPreprocessor from ..db.sql_preprocessor import SQLPreprocessor
from ..errors import UsageError from ..errors import UsageError
from ..db.query_pool import QueryPool from ..db.query_pool import QueryPool
from ..data.place_info import PlaceInfo from ..data.place_info import PlaceInfo
from ..tokenizer.base import AbstractTokenizer from ..tokenizer.base import AbstractTokenizer
from . import freeze
LOG = logging.getLogger() LOG = logging.getLogger()
@@ -89,19 +90,16 @@ async def add_tiger_data(data_dir: str, config: Configuration, threads: int,
""" """
dsn = config.get_libpq_dsn() dsn = config.get_libpq_dsn()
with connect(dsn) as conn:
if freeze.is_frozen(conn):
raise UsageError("Tiger cannot be imported when database frozen (Github issue #3048)")
with TigerInput(data_dir) as tar: with TigerInput(data_dir) as tar:
if not tar: if not tar:
return 1 return 1
with connect(dsn) as conn: with connect(dsn) as conn:
sql = SQLPreprocessor(conn, config) sql = SQLPreprocessor(conn, config)
if not table_exists(conn, 'search_name'):
raise UsageError(
"Cannot perform tiger import: required tables are missing. "
"See https://github.com/osm-search/Nominatim/issues/2463 for details."
)
sql.run_sql_file(conn, 'tiger_import_start.sql') sql.run_sql_file(conn, 'tiger_import_start.sql')
# Reading files and then for each file line handling # Reading files and then for each file line handling

View File

@@ -55,7 +55,7 @@ def parse_version(version: str) -> NominatimVersion:
return NominatimVersion(*[int(x) for x in parts[:2] + parts[2].split('-')]) return NominatimVersion(*[int(x) for x in parts[:2] + parts[2].split('-')])
NOMINATIM_VERSION = parse_version('5.2.99-2') NOMINATIM_VERSION = parse_version('5.2.0-0')
POSTGRESQL_REQUIRED_VERSION = (12, 0) POSTGRESQL_REQUIRED_VERSION = (12, 0)
POSTGIS_REQUIRED_VERSION = (3, 0) POSTGIS_REQUIRED_VERSION = (3, 0)

View File

@@ -9,7 +9,6 @@ Fixtures for BDD test steps
""" """
import sys import sys
import json import json
import re
from pathlib import Path from pathlib import Path
import psycopg import psycopg
@@ -21,8 +20,7 @@ sys.path.insert(0, str(SRC_DIR / 'src'))
import pytest import pytest
from pytest_bdd.parsers import re as step_parse from pytest_bdd.parsers import re as step_parse
from pytest_bdd import given, when, then, scenario from pytest_bdd import given, when, then
from pytest_bdd.feature import get_features
pytest.register_assert_rewrite('utils') pytest.register_assert_rewrite('utils')
@@ -375,57 +373,3 @@ def check_place_missing_lines(db_conn, table, osm_type, osm_id, osm_class):
with db_conn.cursor() as cur: with db_conn.cursor() as cur:
assert cur.execute(sql, params).fetchone()[0] == 0 assert cur.execute(sql, params).fetchone()[0] == 0
if pytest.version_tuple >= (8, 0, 0):
def pytest_pycollect_makemodule(module_path, parent):
return BddTestCollector.from_parent(parent, path=module_path)
class BddTestCollector(pytest.Module):
def __init__(self, **kwargs):
super().__init__(**kwargs)
def collect(self):
for item in super().collect():
yield item
if hasattr(self.obj, 'PYTEST_BDD_SCENARIOS'):
for path in self.obj.PYTEST_BDD_SCENARIOS:
for feature in get_features([str(Path(self.path.parent, path).resolve())]):
yield FeatureFile.from_parent(self,
name=str(Path(path, feature.rel_filename)),
path=Path(feature.filename),
feature=feature)
# borrowed from pytest-bdd: src/pytest_bdd/scenario.py
def make_python_name(string: str) -> str:
"""Make python attribute name out of a given string."""
string = re.sub(r"\W", "", string.replace(" ", "_"))
return re.sub(r"^\d+_*", "", string).lower()
class FeatureFile(pytest.File):
class obj:
pass
def __init__(self, feature, **kwargs):
self.feature = feature
super().__init__(**kwargs)
def collect(self):
for sname, sobject in self.feature.scenarios.items():
class_name = f"L{sobject.line_number}"
test_name = "test_" + make_python_name(sname)
@scenario(self.feature.filename, sname)
def _test():
pass
tclass = type(class_name, (),
{test_name: staticmethod(_test)})
setattr(self.obj, class_name, tclass)
yield pytest.Class.from_parent(self, name=class_name, obj=tclass)

View File

@@ -42,22 +42,6 @@ Feature: Tests for finding places by osm_type and osm_id
| jsonv2 | json | | jsonv2 | json |
| geojson | geojson | | geojson | geojson |
Scenario Outline: Lookup with entrances
When sending v1/lookup with format <format>
| osm_ids | entrances |
| W429210603 | 1 |
Then a HTTP 200 is returned
And the result is valid <outformat>
And result 0 contains in field entrances+0
| osm_id | type | lat | lon |
| 6580031131 | yes | 47.2489382 | 9.5284033 |
Examples:
| format | outformat |
| json | json |
| jsonv2 | json |
| geojson | geojson |
Scenario: Linked places return information from the linkee Scenario: Linked places return information from the linkee
When sending v1/lookup with format geocodejson When sending v1/lookup with format geocodejson
| osm_ids | | osm_ids |

View File

@@ -167,18 +167,3 @@ Feature: v1/reverse Parameter Tests
| json | json | | json | json |
| jsonv2 | json | | jsonv2 | json |
| xml | xml | | xml | xml |
Scenario Outline: Reverse with entrances
When sending v1/reverse with format <format>
| lat | lon | entrances | zoom |
| 47.24942041089678 | 9.52854573737568 | 1 | 18 |
Then a HTTP 200 is returned
And the result is valid <outformat>
And the result contains array field entrances where element 0 contains
| osm_id | type | lat | lon |
| 6580031131 | yes | 47.2489382 | 9.5284033 |
Examples:
| format | outformat |
| json | json |
| jsonv2 | json |

View File

@@ -268,6 +268,33 @@ Feature: Address computation
| W93 | R34 | | W93 | R34 |
| W93 | R4 | | W93 | R4 |
Scenario: postcode boundaries do appear in the address of a way
Given the grid with origin DE
| 1 | | | | | 8 | | 6 | | 2 |
| |10 |11 | | | | | | | |
| |13 |12 | | | | | | | |
| 20| | | 21| | | | | | |
| | | | | | | | | | |
| | | | | | 9 | | | | |
| 4 | | | | | | | 7 | | 3 |
And the named places
| osm | class | type | admin | addr+postcode | geometry |
| R1 | boundary | administrative | 6 | 10000 | (1,2,3,4,1) |
| R34 | boundary | administrative | 8 | 11000 | (1,6,7,4,1) |
And the places
| osm | class | type | addr+postcode | geometry |
| R4 | boundary | postal_code | 11200 | (1,8,9,4,1) |
And the named places
| osm | class | type | geometry |
| W93 | highway | residential | 20,21 |
And the places
| osm | class | type | addr+postcode | geometry |
| W22 | place | postcode | 11234 | (10,11,12,13,10) |
When importing
Then place_addressline contains
| object | address |
| W93 | R4 |
Scenario: squares do not appear in the address of a street Scenario: squares do not appear in the address of a street
Given the grid Given the grid
| | 1 | | 2 | | | | 1 | | 2 | |

View File

@@ -16,6 +16,21 @@ Feature: Linking of places
| R13 | - | | R13 | - |
| N256 | - | | N256 | - |
Scenario: Postcode areas cannot be linked
Given the grid with origin US
| 1 | | 2 |
| | 9 | |
| 4 | | 3 |
And the named places
| osm | class | type | addr+postcode | extra+wikidata | geometry |
| R13 | boundary | postal_code | 12345 | Q87493 | (1,2,3,4,1) |
| N25 | place | suburb | 12345 | Q87493 | 9 |
When importing
Then placex contains
| object | linked_place_id |
| R13 | - |
| N25 | - |
Scenario: Waterways are linked when in waterway relations Scenario: Waterways are linked when in waterway relations
Given the grid Given the grid
| 1 | | | | 3 | 4 | | | | 6 | | 1 | | | | 3 | 4 | | | | 6 |
@@ -297,30 +312,20 @@ Feature: Linking of places
| R1 | LabelPlace | | R1 | LabelPlace |
@skip
Scenario: Linked places expand default language names Scenario: Linked places expand default language names
Given the grid with origin CO Given the grid
| 1 | | 2 | | 5 | | 6 | | 1 | | 2 |
| | 9 | | | | 10 | | | | 9 | |
| 4 | | 3 | | 8 | | 7 | | 4 | | 3 |
And the places Given the places
| osm | class | type | name+name | geometry | | osm | class | type | name+name | geometry |
| N9 | place | city | Popayán | 9 | | N9 | place | city | Popayán | 9 |
And the places | R1 | boundary | administrative | Perímetro Urbano Popayán | (1,2,3,4,1) |
| osm | class | type | name+name:en | geometry |
| N10 | place | city | Open | 10 |
And the places
| osm | class | type | name+name | geometry | admin |
| R1 | boundary | administrative | Perímetro Urbano Popayán | (1,2,3,4,1) | 8 |
| R2 | boundary | administrative | Abre | (5,6,7,8,5) | 8 |
And the relations And the relations
| id | members | | id | members |
| 1 | N9:label | | 1 | N9:label |
| 2 | N10:label |
When importing When importing
Then placex contains
| object | linked_place_id |
| N9:place | R1 |
| R1:boundary | - |
Then placex contains Then placex contains
| object | name+_place_name | name+_place_name:es | | object | name+_place_name | name+_place_name:es |
| R1 | Popayán | Popayán | | R1 | Popayán | Popayán |

Some files were not shown because too many files have changed in this diff Show More