prepare release 3.7.1

Not supported by Python 3.5.

.github/ISSUE_TEMPLATE/feature_request.md

@@ -7,16 +7,16 @@ assignees: ''
 
 ---
 
-Before opening a new feature request, please search through the open issue to check that your request hasn't been reported already.
+<!-- Before opening a new feature request, please search through the open issue to check that your request hasn't been reported already. -->
 
 **Is your feature request related to a problem? Please describe.**
-A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
+<!-- A clear and concise description of what the problem is. Ex. I'm always frustrated when [...] -->
 
 **Describe the solution you'd like**
-A clear and concise description of what you want to happen.
+<!-- A clear and concise description of what you want to happen. -->
 
 **Describe alternatives you've considered**
-A clear and concise description of any alternative solutions or features you've considered.
+<!-- A clear and concise description of any alternative solutions or features you've considered. -->
 
 **Additional context**
-Add any other context or screenshots about the feature request here.
+<!-- Add any other context or screenshots about the feature request here. -->

.github/ISSUE_TEMPLATE/report-issues-with-search-results.md

@@ -9,7 +9,7 @@ assignees: ''
 
 ## What did you search for?
 
-Please try to provide a link to your search. You  can go to https://nominatim.openstreetmap.org and repeat your search there. If you originally found the issue somewhere else, please tell us what software/website you were using.
+<!-- Please try to provide a link to your search. You  can go to https://nominatim.openstreetmap.org and repeat your search there. If you originally found the issue somewhere else, please tell us what software/website you were using. -->
 
 ## What result did you get?
 
@@ -17,11 +17,11 @@ Please try to provide a link to your search. You  can go to https://nominatim.op
 
 **Is the result in the right place and just named wrongly?** 
 
-Please tell us the display name you expected.
+<!-- Please tell us the display name you expected. -->
 
 **Is the result missing completely?**
 
-Make sure that the data you are looking for is in OpenStreetMap. Provide a link to the OpenStreetMap object or if you cannot get it, a link to the map on https://openstreetmap.org where you expect the result to be.
+<!-- Make sure that the data you are looking for is in OpenStreetMap. Provide a link to the OpenStreetMap object or if you cannot get it, a link to the map on https://openstreetmap.org where you expect the result to be.
 
 To get the link to the OSM object, you can try the following:
 
@@ -30,7 +30,8 @@ To get the link to the OSM object, you can try the following:
  * Click on the question mark on the right side of the map. You get a question cursor. Use it to click on the map where your object is located.
  * Find the object of interest in the list that appears on the left side.
  * Click on the object and report back the URL that the browser shows.
+-->
 
 ## Further details
 
-Anything else we should know about the search. Particularities with addresses in the area etc.
+<!-- Anything else we should know about the search. Particularities with addresses in the area etc. -->

.github/ISSUE_TEMPLATE/report-problems-with-the-software.md

@@ -7,13 +7,13 @@ assignees: ''
 
 ---
 
-___Note: if you are installing Nominatim through a docker image, you should report issues with the installation process with the docker repository first.___
+<!-- Note: if you are installing Nominatim through a docker image, you should report issues with the installation process with the docker repository first. -->
 
 **Describe the bug**
-A clear and concise description of what the bug is.
+<!-- A clear and concise description of what the bug is. -->
 
 **To Reproduce**
-Please describe what you did to get to the issue.
+<!-- Please describe what you did to get to the issue. -->
 
 **Software Environment (please complete the following information):**
 - Nominatim version: 
@@ -27,5 +27,10 @@ Please describe what you did to get to the issue.
 - type and size of disks:
 - bare metal/AWS/other cloud service: 
 
+**Postgresql Configuration:**
+
+<!-- List any configuration items you changed in your postgresql configuration. -->
+
 **Additional context**
-Add any other context about the problem here.
+
+<!-- Add any other context about the problem here. -->

.github/actions/build-nominatim/action.yml

@@ -4,25 +4,26 @@ runs:
     using: "composite"
 
     steps:
-        - name: Install prerequisits
-          run: sudo apt-get install -y -qq libboost-system-dev libboost-filesystem-dev libexpat1-dev zlib1g-dev libbz2-dev libpq-dev libproj-dev python3-psycopg2 python3-pyosmium
+        - name: Install prerequisites
+          run: |
+            sudo apt-get install -y -qq libboost-system-dev libboost-filesystem-dev libexpat1-dev zlib1g-dev libbz2-dev libpq-dev libproj-dev libicu-dev python3-psycopg2 python3-pyosmium python3-dotenv python3-psutil python3-jinja2 python3-icu
+          shell: bash
+
+        - name: Download dependencies
+          run: |
+              if [ ! -f country_grid.sql.gz ]; then
+                  wget --no-verbose https://www.nominatim.org/data/country_grid.sql.gz
+              fi
+              cp country_grid.sql.gz Nominatim/data/country_osm_grid.sql.gz
           shell: bash
 
         - name: Configure
-          run: mkdir build && cd build && cmake ..
+          run: mkdir build && cd build && cmake ../Nominatim
           shell: bash
 
         - name: Build
           run: |
               make -j2 all
-              ./utils/setup.php --setup-website
+              sudo make install
           shell: bash
           working-directory: build
-
-        - name: Download dependencies
-          run: |
-              if [ ! -f data/country_osm_grid.sql.gz ]; then
-                  wget --no-verbose -O data/country_osm_grid.sql.gz https://www.nominatim.org/data/country_grid.sql.gz
-              fi
-          shell: bash
-

.github/actions/setup-postgresql/action.yml

@@ -14,8 +14,10 @@ runs:
     steps:
         - name: Remove existing PostgreSQL
           run: |
-              sudo apt-get update -qq
               sudo apt-get purge -yq postgresql*
+              sudo sh -c 'echo "deb http://apt.postgresql.org/pub/repos/apt $(lsb_release -cs)-pgdg main" > /etc/apt/sources.list.d/pgdg.list'
+              sudo apt-get update -qq
+
           shell: bash
 
         - name: Install PostgreSQL

.github/workflows/ci-tests.yml

@@ -19,6 +19,13 @@ jobs:
             - uses: actions/checkout@v2
               with:
                   submodules: true
+                  path: Nominatim
+
+            - name: Setup PHP
+              uses: shivammathur/setup-php@v2
+              with:
+                  php-version: '7.4'
+                  tools: phpunit, phpcs
 
             - name: Get Date
               id: get-date
@@ -29,39 +36,57 @@ jobs:
             - uses: actions/cache@v2
               with:
                   path: |
-                      data/country_osm_grid.sql.gz
-                      monaco-latest.osm.pbf
-                  key: nominatim-data-${{ steps.get-date.outputs.date }}
+                     country_grid.sql.gz
+                  key: nominatim-country-data-${{ steps.get-date.outputs.date }}
 
-            - uses: ./.github/actions/setup-postgresql
+            - uses: ./Nominatim/.github/actions/setup-postgresql
               with:
                   postgresql-version: ${{ matrix.postgresql }}
                   postgis-version: ${{ matrix.postgis }}
-            - uses: ./.github/actions/build-nominatim
+            - uses: ./Nominatim/.github/actions/build-nominatim
 
             - name: Install test prerequsites
-              run: |
-                   sudo apt-get install -y -qq php-codesniffer python3-tidylib
-                   sudo pip3 install behave nose
+              run: sudo apt-get install -y -qq php-codesniffer pylint python3-pytest python3-behave
 
             - name: PHP linting
               run: phpcs --report-width=120 .
+              working-directory: Nominatim
+
+            - name: Python linting
+              run: pylint --extension-pkg-whitelist=osmium nominatim
+              working-directory: Nominatim
 
             - name: PHP unit tests
               run: phpunit ./
-              working-directory: test/php
+              working-directory: Nominatim/test/php
+
+            - name: Python unit tests
+              run: py.test-3 test/python
+              working-directory: Nominatim
 
             - name: BDD tests
-              run: behave -DREMOVE_TEMPLATE=1 --format=progress3 db osm2pgsql
-              working-directory: test/bdd
+              run: behave -DREMOVE_TEMPLATE=1 -DBUILDDIR=$GITHUB_WORKSPACE/build --format=progress3
+              working-directory: Nominatim/test/bdd
 
     import:
-        runs-on: ubuntu-20.04
+        strategy:
+            matrix:
+                ubuntu: [18, 20]
+                include:
+                    - ubuntu: 18
+                      postgresql: 9.5
+                      postgis: 2.5
+                    - ubuntu: 20
+                      postgresql: 13
+                      postgis: 3
+
+        runs-on: ubuntu-${{ matrix.ubuntu }}.04
 
         steps:
             - uses: actions/checkout@v2
               with:
                   submodules: true
+                  path: Nominatim
 
             - name: Get Date
               id: get-date
@@ -72,49 +97,71 @@ jobs:
             - uses: actions/cache@v2
               with:
                   path: |
-                      data/country_osm_grid.sql.gz
-                      monaco-latest.osm.pbf
-                  key: nominatim-data-${{ steps.get-date.outputs.date }}
+                     country_grid.sql.gz
+                  key: nominatim-country-data-${{ steps.get-date.outputs.date }}
 
-            - uses: ./.github/actions/setup-postgresql
+            - uses: actions/cache@v2
               with:
-                  postgresql-version: 13
-                  postgis-version: 3
-            - uses: ./.github/actions/build-nominatim
+                  path: |
+                     monaco-latest.osm.pbf
+                  key: nominatim-test-data-${{ steps.get-date.outputs.date }}
 
-            - name: Create configuration
+            - uses: actions/setup-python@v2
+              with:
+                python-version: 3.5
+              if: matrix.ubuntu == 18
+
+            - uses: ./Nominatim/.github/actions/setup-postgresql
+              with:
+                  postgresql-version: ${{ matrix.postgresql }}
+                  postgis-version: ${{ matrix.postgis }}
+            - uses: ./Nominatim/.github/actions/build-nominatim
+
+            - name: Install extra dependencies for Ubuntu 18
               run: |
-                   echo '<?php' > settings/local.php
-                   echo " @define('CONST_Pyosmium_Binary', '/usr/lib/python3-pyosmium/pyosmium-get-changes');" >> settings/local.php
-              working-directory: build
+                sudo apt-get install libicu-dev
+                pip3 install python-dotenv psycopg2==2.7.7 jinja2==2.8 psutil==5.4.2 pyicu osmium
+              shell: bash
+              if: matrix.ubuntu == 18
 
-            - name: Download import data
+            - name: Clean installation
+              run: rm -rf Nominatim build
+              shell: bash
+
+            - name: Prepare import environment
               run: |
                   if [ ! -f monaco-latest.osm.pbf ]; then
                       wget --no-verbose https://download.geofabrik.de/europe/monaco-latest.osm.pbf
                   fi
+                  mkdir data-env
+                  cd data-env
               shell: bash
 
             - name: Import
-              run: php ./utils/setup.php --osm-file ../monaco-latest.osm.pbf --osm2pgsql-cache 500 --all
-              working-directory: build
+              run: nominatim import --osm-file ../monaco-latest.osm.pbf
+              shell: bash
+              working-directory: data-env
 
             - name: Import special phrases
-              run: php ./utils/specialphrases.php --wiki-import | psql -d nominatim
-              working-directory: build
+              run: nominatim special-phrases --import-from-wiki
+              working-directory: data-env
 
             - name: Check import
-              run: php ./utils/check_import_finished.php
-              working-directory: build
+              run: nominatim admin --check-database
+              working-directory: data-env
+
+            - name: Warm up database
+              run: nominatim admin --warm
+              working-directory: data-env
 
             - name: Run update
               run: |
-                   php ./utils/update.php --init-updates
-                   php ./utils/update.php --import-osmosis
-              working-directory: build
+                   nominatim replication --init
+                   nominatim replication --once
+              working-directory: data-env
 
             - name: Run reverse-only import
-              run : |
-                  dropdb nominatim
-                  php ./utils/setup.php --osm-file ../monaco-latest.osm.pbf --reverse-only --all
-              working-directory: build
+              run : nominatim import --osm-file ../monaco-latest.osm.pbf --reverse-only
+              working-directory: data-env
+              env:
+                  NOMINATIM_DATABASE_DSN: pgsql:dbname=reverse

.gitignore

@@ -9,3 +9,4 @@ data/wiki_specialphrases.sql
 data/osmosischange.osc
 
 .vagrant
+data/country_osm_grid.sql.gz

.pylintrc

@@ -0,0 +1,12 @@
+[MASTER]
+
+extension-pkg-whitelist=osmium
+ignored-modules=icu
+
+[MESSAGES CONTROL]
+
+[TYPECHECK]
+
+# closing added here because it sometimes triggers a false positive with
+# 'with' statements.
+ignored-classes=NominatimArgs,closing

CMakeLists.txt

@@ -6,7 +6,7 @@
 #
 #-----------------------------------------------------------------------------
 
-cmake_minimum_required(VERSION 2.8 FATAL_ERROR)
+cmake_minimum_required(VERSION 3.0 FATAL_ERROR)
 list(APPEND CMAKE_MODULE_PATH "${CMAKE_SOURCE_DIR}/cmake")
 
 
@@ -19,8 +19,8 @@ list(APPEND CMAKE_MODULE_PATH "${CMAKE_SOURCE_DIR}/cmake")
 project(nominatim)
 
 set(NOMINATIM_VERSION_MAJOR 3)
-set(NOMINATIM_VERSION_MINOR 6)
-set(NOMINATIM_VERSION_PATCH 0)
+set(NOMINATIM_VERSION_MINOR 7)
+set(NOMINATIM_VERSION_PATCH 1)
 
 set(NOMINATIM_VERSION "${NOMINATIM_VERSION_MAJOR}.${NOMINATIM_VERSION_MINOR}.${NOMINATIM_VERSION_PATCH}")
 
@@ -36,6 +36,7 @@ set(BUILD_API on CACHE BOOL "Build everything for the API server")
 set(BUILD_MODULE on CACHE BOOL "Build PostgreSQL module")
 set(BUILD_TESTS on CACHE BOOL "Build test suite")
 set(BUILD_DOCS on CACHE BOOL "Build documentation")
+set(BUILD_MANPAGE on CACHE BOOL "Build Manual Page")
 set(BUILD_OSM2PGSQL on CACHE BOOL "Build osm2pgsql (expert only)")
 
 #-----------------------------------------------------------------------------
@@ -57,20 +58,11 @@ endif()
 
 
 #-----------------------------------------------------------------------------
-#  python and pyosmium (imports/updates only)
+#  python (imports/updates only)
 #-----------------------------------------------------------------------------
 
 if (BUILD_IMPORTER)
-    find_package(PythonInterp 3)
-
-    find_program(PYOSMIUM pyosmium-get-changes)
-    if (NOT EXISTS "${PYOSMIUM}")
-        set(PYOSMIUM_PATH "")
-            message(WARNING "pyosmium-get-changes not found (required for updates)")
-    else()
-        set(PYOSMIUM_PATH "${PYOSMIUM}")
-        message(STATUS "Using pyosmium-get-changes at ${PYOSMIUM_PATH}")
-    endif()
+    find_package(PythonInterp 3.5 REQUIRED)
 endif()
 
 #-----------------------------------------------------------------------------
@@ -86,8 +78,19 @@ if (BUILD_API OR BUILD_IMPORTER)
     # sanity check if PHP binary exists
     if (NOT EXISTS ${PHP_BIN})
         message(FATAL_ERROR "PHP binary not found. Install php or provide location with -DPHP_BIN=/path/php ")
+    else()
+        message (STATUS "Using PHP binary " ${PHP_BIN})
+    endif()
+    if (NOT PHPCGI_BIN)
+        find_program (PHPCGI_BIN php-cgi)
+    endif()
+    # sanity check if PHP binary exists
+    if (NOT EXISTS ${PHPCGI_BIN})
+        message(WARNING "php-cgi binary not found. nominatim tool will not provide query functions.")
+        set (PHPCGI_BIN "")
+    else()
+        message (STATUS "Using php-cgi binary " ${PHPCGI_BIN})
     endif()
-    message (STATUS "Using PHP binary " ${PHP_BIN})
 endif()
 
 #-----------------------------------------------------------------------------
@@ -95,70 +98,36 @@ endif()
 #-----------------------------------------------------------------------------
 
 if (BUILD_IMPORTER)
+   find_file(COUNTRY_GRID_FILE country_osm_grid.sql.gz
+             PATHS ${PROJECT_SOURCE_DIR}/data
+             NO_DEFAULT_PATH
+             DOC "Location of the country grid file."
+            )
+
+   if (NOT COUNTRY_GRID_FILE)
+       message(FATAL_ERROR "\nYou need to download the country_osm_grid first:\n"
+                           "    wget -O ${PROJECT_SOURCE_DIR}/data/country_osm_grid.sql.gz https://www.nominatim.org/data/country_grid.sql.gz")
+   endif()
+
    set(CUSTOMSCRIPTS
-       utils/check_import_finished.php
-       utils/country_languages.php
-       utils/importWikipedia.php
-       utils/export.php
-       utils/query.php
-       utils/setup.php
-       utils/specialphrases.php
-       utils/update.php
-       utils/warm.php
+       check_import_finished.php
+       country_languages.php
+       export.php
+       query.php
+       setup.php
+       update.php
+       warm.php
       )
 
    foreach (script_source ${CUSTOMSCRIPTS})
        configure_file(${PROJECT_SOURCE_DIR}/cmake/script.tmpl
-                      ${PROJECT_BINARY_DIR}/${script_source})
+                      ${PROJECT_BINARY_DIR}/utils/${script_source})
    endforeach()
+
+   configure_file(${PROJECT_SOURCE_DIR}/cmake/tool.tmpl
+                  ${PROJECT_BINARY_DIR}/nominatim)
 endif()
 
-#-----------------------------------------------------------------------------
-# webserver scripts (API only)
-#-----------------------------------------------------------------------------
-
-if (BUILD_API)
-   set(WEBSITESCRIPTS
-       website/deletable.php
-       website/details.php
-       website/lookup.php
-       website/polygons.php
-       website/reverse.php
-       website/search.php
-       website/status.php
-   )
-
-   foreach (script_source ${WEBSITESCRIPTS})
-       configure_file(${PROJECT_SOURCE_DIR}/cmake/website.tmpl
-                      ${PROJECT_BINARY_DIR}/${script_source})
-   endforeach()
-
-   set(WEBPATHS css images js)
-
-   foreach (wp ${WEBPATHS})
-       execute_process(
-           COMMAND ln -sf ${PROJECT_SOURCE_DIR}/website/${wp} ${PROJECT_BINARY_DIR}/website/
-       )
-   endforeach()
-
-   add_custom_target(serve
-           php -S 127.0.0.1:8088
-           WORKING_DIRECTORY ${PROJECT_BINARY_DIR}/website
-   )
-
-   add_custom_target(serve-global
-           php -S 0.0.0.0:8088
-           WORKING_DIRECTORY ${PROJECT_BINARY_DIR}/website
-   )
-endif()
-
-#-----------------------------------------------------------------------------
-# default settings
-#-----------------------------------------------------------------------------
-
-configure_file(${PROJECT_SOURCE_DIR}/settings/defaults.php
-               ${PROJECT_BINARY_DIR}/settings/settings.php)
-
 #-----------------------------------------------------------------------------
 # Tests
 #-----------------------------------------------------------------------------
@@ -168,21 +137,60 @@ if (BUILD_TESTS)
 
     set(TEST_BDD db osm2pgsql api)
 
-    foreach (test ${TEST_BDD})
-        add_test(NAME bdd_${test}
-                 COMMAND behave ${test}
-                 WORKING_DIRECTORY ${PROJECT_SOURCE_DIR}/test/bdd)
-        set_tests_properties(bdd_${test}
-            PROPERTIES ENVIRONMENT "NOMINATIM_DIR=${PROJECT_BINARY_DIR}")
-    endforeach()
+    find_program(PYTHON_BEHAVE behave)
+    find_program(PYLINT NAMES pylint3 pylint)
+    find_program(PYTEST NAMES pytest py.test-3 py.test)
+    find_program(PHPCS phpcs)
+    find_program(PHPUNIT phpunit)
 
-    add_test(NAME php
-             COMMAND phpunit ./
-             WORKING_DIRECTORY ${PROJECT_SOURCE_DIR}/test/php)
+    if (PYTHON_BEHAVE)
+        message(STATUS "Using Python behave binary ${PYTHON_BEHAVE}")
+        foreach (test ${TEST_BDD})
+            add_test(NAME bdd_${test}
+                     COMMAND ${PYTHON_BEHAVE} ${test}
+                     WORKING_DIRECTORY ${PROJECT_SOURCE_DIR}/test/bdd)
+            set_tests_properties(bdd_${test}
+                PROPERTIES ENVIRONMENT "NOMINATIM_DIR=${PROJECT_BINARY_DIR}")
+        endforeach()
+    else()
+        message(WARNING "behave not found. BDD tests disabled." )
+    endif()
 
-    add_test(NAME phpcs
-             COMMAND phpcs --report-width=120 --colors lib website utils
-             WORKING_DIRECTORY ${PROJECT_SOURCE_DIR})
+    if (PHPUNIT)
+        message(STATUS "Using phpunit binary ${PHPUNIT}")
+        add_test(NAME php
+                 COMMAND ${PHPUNIT} ./
+                 WORKING_DIRECTORY ${PROJECT_SOURCE_DIR}/test/php)
+    else()
+        message(WARNING "phpunit not found. PHP unit tests disabled." )
+    endif()
+
+    if (PHPCS)
+        message(STATUS "Using phpcs binary ${PHPCS}")
+        add_test(NAME phpcs
+                 COMMAND ${PHPCS} --report-width=120 --colors lib website utils
+                 WORKING_DIRECTORY ${PROJECT_SOURCE_DIR})
+    else()
+        message(WARNING "phpcs not found. PHP linting tests disabled." )
+    endif()
+
+    if (PYLINT)
+        message(STATUS "Using pylint binary ${PYLINT}")
+        add_test(NAME pylint
+                 COMMAND ${PYLINT} nominatim
+                 WORKING_DIRECTORY ${PROJECT_SOURCE_DIR})
+    else()
+        message(WARNING "pylint not found. Python linting tests disabled.")
+    endif()
+
+    if (PYTEST)
+        message(STATUS "Using pytest binary ${PYTEST}")
+        add_test(NAME pytest
+                 COMMAND ${PYTEST} test/python
+                 WORKING_DIRECTORY ${PROJECT_SOURCE_DIR})
+    else()
+        message(WARNING "pytest not found. Python tests disabled." )
+    endif()
 endif()
 
 #-----------------------------------------------------------------------------
@@ -200,3 +208,69 @@ endif()
 if (BUILD_DOCS)
    add_subdirectory(docs)
 endif()
+
+#-----------------------------------------------------------------------------
+# Manual page
+#-----------------------------------------------------------------------------
+
+if (BUILD_MANPAGE)
+   add_subdirectory(manual)
+endif()
+
+#-----------------------------------------------------------------------------
+# Installation
+#-----------------------------------------------------------------------------
+
+
+include(GNUInstallDirs)
+set(NOMINATIM_DATADIR ${CMAKE_INSTALL_FULL_DATADIR}/${PROJECT_NAME})
+set(NOMINATIM_LIBDIR ${CMAKE_INSTALL_FULL_LIBDIR}/${PROJECT_NAME})
+set(NOMINATIM_CONFIGDIR ${CMAKE_INSTALL_FULL_SYSCONFDIR}/${PROJECT_NAME})
+
+if (BUILD_IMPORTER)
+    configure_file(${PROJECT_SOURCE_DIR}/cmake/tool-installed.tmpl installed.bin)
+    install(PROGRAMS ${PROJECT_BINARY_DIR}/installed.bin
+            DESTINATION ${CMAKE_INSTALL_BINDIR}
+            RENAME nominatim)
+
+    install(DIRECTORY nominatim
+            DESTINATION ${NOMINATIM_LIBDIR}/lib-python
+            FILES_MATCHING PATTERN "*.py"
+            PATTERN __pycache__ EXCLUDE)
+    install(DIRECTORY lib-sql DESTINATION ${NOMINATIM_LIBDIR})
+
+    install(FILES data/country_name.sql
+                  ${COUNTRY_GRID_FILE}
+                  data/words.sql
+            DESTINATION ${NOMINATIM_DATADIR})
+endif()
+
+if (BUILD_OSM2PGSQL)
+    if (${CMAKE_VERSION} VERSION_LESS 3.13)
+        # Installation of subdirectory targets was only introduced in 3.13.
+        # So just copy the osm2pgsql file for older versions.
+        install(PROGRAMS ${PROJECT_BINARY_DIR}/osm2pgsql/osm2pgsql
+                DESTINATION ${NOMINATIM_LIBDIR})
+    else()
+        install(TARGETS osm2pgsql RUNTIME DESTINATION ${NOMINATIM_LIBDIR})
+    endif()
+endif()
+
+if (BUILD_MODULE)
+    install(PROGRAMS ${PROJECT_BINARY_DIR}/module/nominatim.so
+            DESTINATION ${NOMINATIM_LIBDIR}/module)
+endif()
+
+if (BUILD_API)
+    install(DIRECTORY lib-php DESTINATION ${NOMINATIM_LIBDIR})
+endif()
+
+install(FILES settings/env.defaults
+              settings/address-levels.json
+              settings/phrase-settings.json
+              settings/import-admin.style
+              settings/import-street.style
+              settings/import-address.style
+              settings/import-full.style
+              settings/import-extratags.style
+        DESTINATION ${NOMINATIM_CONFIGDIR})

CONTRIBUTING.md

@@ -49,22 +49,18 @@ are in process of consolidating the style. The following rules apply:
  * for PHP variables use CamelCase with a prefixing letter indicating the type
    (i - integer, f - float, a - array, s - string, o - object)
 
-The coding style is enforced with PHPCS and can be tested with:
+The coding style is enforced with PHPCS and pylint. It can be tested with:
 
 ```
-  phpcs --report-width=120 --colors .
+phpcs --report-width=120 --colors .
+pylint3 --extension-pkg-whitelist=osmium nominatim
 ```
 
 ## Testing
 
-Before submitting a pull request make sure that the following tests pass:
+Before submitting a pull request make sure that the tests pass:
 
 ```
-  cd test/bdd
-  behave -DBUILDDIR=<builddir> db osm2pgsql
-```
-
-```
-  cd test/php
-  phpunit ./
+  cd build
+  make test
 ```

ChangeLog

@@ -1,3 +1,34 @@
+3.7.1
+
+ * fix smaller issues with special phrases import
+ * add index to speed up continued indexing during import
+ * fix index on location_property_tiger(parent_place_id)
+ * make sure Python code is backward-compatible with Python 3.5
+ * various documentation fixes
+
+3.7.0
+
+ * switch to dotenv for configuration file
+ * introduce 'make install' (reorganising most of the code)
+ * introduce nominatim tool as replacement for various php scripts
+ * introduce project directories and allow multiple installations from same build
+ * clean up BDD tests: drop nose, reorganise step code
+ * simplify test database for API BDD tests and autoinstall database
+ * port most of the code for command-line tools to Python
+   (thanks to @darkshredder and @AntoJvlt)
+ * add tests for all tooling
+ * replace pyosmium-get-changes with custom internal implementation using
+   pyosmium
+ * improve search for queries with housenumber and partial terms
+ * add database versioning
+ * use jinja2 for preprocessing SQL files
+ * introduce automatic migrations
+ * reverse fix preference of interpolations over housenumbers
+ * parallelize indexing of postcodes
+ * add non-key indexes to speed up housenumber + street searches
+ * switch housenumber field in placex to save transliterated names
+
+
 3.6.0
 
  * add full support for searching by and displaying of addr:* tags

README.md

@@ -24,15 +24,14 @@ Installing and running Nominatim is something for experienced system
 administrators only who can do some trouble-shooting themselves. We are sorry,
 but we can not provide installation support. We are all doing this in our free
 time and there is just so much of that time to go around. Do not open issues in
-our bug tracker if you need help. You can ask questions on the mailing list
-(see below) or on [help.openstreetmap.org](https://help.openstreetmap.org/).**
+our bug tracker if you need help. Use the discussions forum
+or ask for help on [help.openstreetmap.org](https://help.openstreetmap.org/).**
 
 The latest stable release can be downloaded from https://nominatim.org.
 There you can also find [installation instructions for the release](https://nominatim.org/release-docs/latest/admin/Installation), as well as an extensive [Troubleshooting/FAQ section](https://nominatim.org/release-docs/latest/admin/Faq/).
 
-Detailed installation instructions for the development version can be
-found at [nominatim.org](https://nominatim.org/release-docs/develop/admin/Installation)
-as well.
+[Detailed installation instructions for current master](https://nominatim.org/release-docs/develop/admin/Installation)
+can be found at nominatim.org as well.
 
 A quick summary of the necessary steps:
 
@@ -42,12 +41,15 @@ A quick summary of the necessary steps:
         cd build
         cmake ..
         make
+        sudo make install
 
-2. Get OSM data and import:
+2. Create a project directory, get OSM data and import:
 
-        ./build/utils/setup.php --osm-file <your planet file> --all
+        mkdir nominatim-project
+        cd nominatim-project
+        nominatim import --osm-file <your planet file>
 
-3. Point your webserver to the ./build/website directory.
+3. Point your webserver to the nominatim-project/website directory.
 
 
 License
@@ -59,13 +61,14 @@ The source code is available under a GPLv2 license.
 Contributing
 ============
 
-Contributions are welcome. For details see [contribution guide](CONTRIBUTING.md).
-
-Both bug reports and pull requests are welcome.
+Contributions, bugreport and pull requests are welcome.
+For details see [contribution guide](CONTRIBUTING.md).
 
 
-Mailing list
-============
+Questions and help
+==================
 
-For questions you can join the geocoding mailing list, see
-https://lists.openstreetmap.org/listinfo/geocoding
+For questions, community help and discussions you can use the
+[Github discussions forum](https://github.com/osm-search/Nominatim/discussions)
+or join the
+[geocoding mailing list](https://lists.openstreetmap.org/listinfo/geocoding).

cmake/script.tmpl

@@ -1,4 +1,14 @@
 #!@PHP_BIN@ -Cq
 <?php
-require_once(dirname(dirname(__FILE__)).'/settings/settings.php');
-require_once(CONST_BasePath.'/@script_source@');
+require('@CMAKE_SOURCE_DIR@/lib-php/dotenv_loader.php');
+
+@define('CONST_Default_ModulePath', '@CMAKE_BINARY_DIR@/module');
+@define('CONST_Default_Osm2pgsql', '@CMAKE_BINARY_DIR@/osm2pgsql/osm2pgsql');
+@define('CONST_DataDir', '@CMAKE_SOURCE_DIR@/data');
+@define('CONST_SqlDir', '@CMAKE_SOURCE_DIR@/lib-sql');
+@define('CONST_ConfigDir', '@CMAKE_SOURCE_DIR@/settings');
+
+loadDotEnv();
+$_SERVER['NOMINATIM_NOMINATIM_TOOL'] = '@CMAKE_BINARY_DIR@/nominatim';
+
+require_once('@CMAKE_SOURCE_DIR@/lib-php/admin/@script_source@');

cmake/tool-installed.tmpl

@@ -0,0 +1,17 @@
+#!/usr/bin/env python3
+import sys
+import os
+
+sys.path.insert(1, '@NOMINATIM_LIBDIR@/lib-python')
+
+os.environ['NOMINATIM_NOMINATIM_TOOL'] = os.path.abspath(__file__)
+
+from nominatim import cli
+
+exit(cli.nominatim(module_dir='@NOMINATIM_LIBDIR@/module',
+                   osm2pgsql_path='@NOMINATIM_LIBDIR@/osm2pgsql',
+                   phplib_dir='@NOMINATIM_LIBDIR@/lib-php',
+                   sqllib_dir='@NOMINATIM_LIBDIR@/lib-sql',
+                   data_dir='@NOMINATIM_DATADIR@',
+                   config_dir='@NOMINATIM_CONFIGDIR@',
+                   phpcgi_path='@PHPCGI_BIN@'))

cmake/tool.tmpl

@@ -0,0 +1,17 @@
+#!/usr/bin/env python3
+import sys
+import os
+
+sys.path.insert(1, '@CMAKE_SOURCE_DIR@')
+
+os.environ['NOMINATIM_NOMINATIM_TOOL'] = os.path.abspath(__file__)
+
+from nominatim import cli
+
+exit(cli.nominatim(module_dir='@CMAKE_BINARY_DIR@/module',
+                   osm2pgsql_path='@CMAKE_BINARY_DIR@/osm2pgsql/osm2pgsql',
+                   phplib_dir='@CMAKE_SOURCE_DIR@/lib-php',
+                   sqllib_dir='@CMAKE_SOURCE_DIR@/lib-sql',
+                   data_dir='@CMAKE_SOURCE_DIR@/data',
+                   config_dir='@CMAKE_SOURCE_DIR@/settings',
+                   phpcgi_path='@PHPCGI_BIN@'))

cmake/website.tmpl

@@ -1,5 +0,0 @@
-<?php
-@define('CONST_Debug', (isset($_GET['debug']) && $_GET['debug']));
-require_once(dirname(dirname(__FILE__)).'/settings/settings-frontend.php');
-
-require_once(CONST_BasePath.'/@script_source@');

data/gb_postcode_table.sql

@@ -1,26 +0,0 @@
--- This data contains Ordnance Survey data © Crown copyright and database right 2010. 
--- Code-Point Open contains Royal Mail data © Royal Mail copyright and database right 2010.
--- OS data may be used under the terms of the OS OpenData licence:
--- http://www.ordnancesurvey.co.uk/oswebsite/opendata/licence/docs/licence.pdf
-
-SET statement_timeout = 0;
-SET client_encoding = 'UTF8';
-SET standard_conforming_strings = off;
-SET check_function_bodies = false;
-SET client_min_messages = warning;
-SET escape_string_warning = off;
-
-SET search_path = public, pg_catalog;
-
-SET default_tablespace = '';
-
-SET default_with_oids = false;
-
-CREATE TABLE gb_postcode (
-    id integer,
-    postcode character varying(9),
-    geometry geometry,
-    CONSTRAINT enforce_dims_geometry CHECK ((st_ndims(geometry) = 2)),
-    CONSTRAINT enforce_srid_geometry CHECK ((st_srid(geometry) = 4326))
-);
-

data/us_postcode_table.sql

@@ -1,16 +0,0 @@
-SET statement_timeout = 0;
-SET client_encoding = 'UTF8';
-SET check_function_bodies = false;
-SET client_min_messages = warning;
-
-SET search_path = public, pg_catalog;
-
-SET default_tablespace = '';
-
-SET default_with_oids = false;
-
-CREATE TABLE us_postcode (
-    postcode text,
-    x double precision,
-    y double precision
-);

docs/admin/Advanced-Installations.md

@@ -47,15 +47,16 @@ The file `import_multiple_regions.sh` needs to be edited as per your requirement
     
         BASEURL="https://download.geofabrik.de"
         DOWNCOUNTRYPOSTFIX="-latest.osm.pbf"
- 
+
+### Setting up multiple regions
+
 !!! tip
-    If your database already exists and you want to add more countries, replace the setting up part
+    If your database already exists and you want to add more countries,
+    replace the setting up part
     `${SETUPFILE} --osm-file ${UPDATEDIR}/tmp/combined.osm.pbf --all 2>&1`
     with `${UPDATEFILE} --import-file ${UPDATEDIR}/tmp/combined.osm.pbf --index --index-instances N 2>&1`
     where N is the numbers of CPUs in your system.
 
-### Setting up multiple regions
-
 Run the following command from your Nominatim directory after configuring the file.
 
     bash ./utils/import_multiple_regions.sh
@@ -154,7 +155,7 @@ Make sure that the PostgreSQL server package is installed on the machine
 the PostgreSQL server itself.
 
 Download and compile Nominatim as per standard instructions. Once done, you find
-the nomrmalization library in `build/module/nominatim.so`. Copy the file to
+the normalization library in `build/module/nominatim.so`. Copy the file to
 the database server at a location where it is readable and executable by the
 PostgreSQL server process.
 
@@ -162,11 +163,11 @@ PostgreSQL server process.
 
 On the client side you now need to configure the import to point to the
 correct location of the library **on the database server**. Add the following
-line to your your `settings/local.php` file:
+line to your your `.env` file:
 
 ```php
-@define('CONST_Database_Module_Path', '<directory on the database server where nominatim.so resides>');
+NOMINATIM_DATABASE_MODULE_PATH="<directory on the database server where nominatim.so resides>"
 ```
 
-Now change the `CONST_Database_DSN` to point to your remote server and continue
+Now change the `NOMINATIM_DATABASE_DSN` to point to your remote server and continue
 to follow the [standard instructions for importing](/admin/Import).

docs/admin/Deployment.md

@@ -1,7 +1,7 @@
 # Deploying Nominatim
 
 The Nominatim API is implemented as a PHP application. The `website/` directory
-in the build directory contains the configured website. You can serve this
+in the project directory contains the configured website. You can serve this
 in a production environment with any web server that is capable to run
 PHP scripts.
 
@@ -13,10 +13,11 @@ to run a web service. Please refer to the documentation of
 for background information on configuring the services.
 
 !!! Note
-    Throughout this page, we assume that your Nominatim build directory is
-    located in `/srv/nominatim/build` and the source code in
-    `/srv/nominatim/Nominatim`. If you have put it somewhere else, you
-    need to adjust the commands and configuration accordingly.
+    Throughout this page, we assume that your Nominatim project directory is
+    located in `/srv/nominatim-project` and that you have installed Nominatim
+    using the default installation prefix `/usr/local`. If you have put it
+    somewhere else, you need to adjust the commands and configuration
+    accordingly.
 
     We further assume that your web server runs as user `www-data`. Older
     versions of CentOS may still use the user name `apache`. You also need
@@ -29,7 +30,7 @@ web server user. You can check that the permissions are correct by accessing
 on of the php files as the web server user:
 
 ``` sh
-sudo -u www-data head -n 1 /srv/nominatim/build/website/search.php
+sudo -u www-data head -n 1 /srv/nominatim-project/website/search.php
 ```
 
 If this shows a permission error, then you need to adapt the permissions of
@@ -40,11 +41,11 @@ web server access. At a minimum the following SELinux labelling should be done
 for Nominatim:
 
 ``` sh
-sudo semanage fcontext -a -t httpd_sys_content_t "/srv/nominatim/Nominatim/(website|lib|settings)(/.*)?"
-sudo semanage fcontext -a -t httpd_sys_content_t "/srv/nominatim/build/(website|settings)(/.*)?"
-sudo semanage fcontext -a -t lib_t "/srv/nominatim/build/module/nominatim.so"
-sudo restorecon -R -v /srv/nominatim/Nominatim
-sudo restorecon -R -v /srv/nominatim/build
+sudo semanage fcontext -a -t httpd_sys_content_t "/usr/local/nominatim/lib/lib-php(/.*)?"
+sudo semanage fcontext -a -t httpd_sys_content_t "/srv/nominatim-project/website(/.*)?"
+sudo semanage fcontext -a -t lib_t "/srv/nominatim-project/module/nominatim.so"
+sudo restorecon -R -v /usr/local/lib/nominatim
+sudo restorecon -R -v /srv/nominatim-project
 ```
 
 ## Nominatim with Apache
@@ -65,13 +66,13 @@ Make sure your Apache configuration contains the required permissions for the
 directory and create an alias:
 
 ``` apache
-<Directory "/srv/nominatim/build/website">
+<Directory "/srv/nominatim-project/website">
   Options FollowSymLinks MultiViews
   AddType text/html   .php
   DirectoryIndex search.php
   Require all granted
 </Directory>
-Alias /nominatim /srv/nominatim/build/website
+Alias /nominatim /srv/nominatim-project/website
 ```
 
 After making changes in the apache config you need to restart apache.
@@ -110,7 +111,7 @@ Tell nginx that php files are special and to fastcgi_pass to the php-fpm
 unix socket by adding the location definition to the default configuration.
 
 ``` nginx
-root /srv/nominatim/build/website;
+root /srv/nominatim-project/website;
 index search.php;
 location / {
     try_files $uri $uri/ @php;

docs/admin/Faq.md

@@ -16,7 +16,7 @@ was killed. If it looks like this:
 then you can resume with the following command:
 
 ```sh
-./utils/setup.php --index --create-search-indices --create-country-names
+nominatim import --continue indexing
 ```
 
 If the reported rank is 26 or higher, you can also safely add `--index-noanalyse`.
@@ -31,7 +31,7 @@ list for hints.
 If it happened during index creation you can try rerunning the step with
 
 ```sh
-./utils/setup.php --create-search-indices --ignore-errors
+nominatim import --continue indexing
 ```
 
 Otherwise it's best to start the full setup from the beginning.
@@ -93,7 +93,7 @@ on a non-managed machine.
 
 ### I see the error: "function transliteration(text) does not exist"
 
-Reinstall the nominatim functions with `setup.php --create--functions`
+Reinstall the nominatim functions with `nominatim refresh --functions`
 and check for any errors, e.g. a missing `nominatim.so` file.
 
 ### I see the error: "ERROR: mmap (remap) failed"
@@ -113,7 +113,8 @@ Double-check clang is installed. Instead of `make` try running `make CLANG=true`
 
 ### nominatim UPDATE failed: ERROR: buffer 179261 is not owned by resource owner Portal
 
-Several users [reported this](https://github.com/openstreetmap/Nominatim/issues/1168) during the initial import of the database. It's
+Several users [reported this](https://github.com/openstreetmap/Nominatim/issues/1168)
+during the initial import of the database. It's
 something PostgreSQL internal Nominatim doesn't control. And PostgreSQL forums
 suggest it's threading related but definitely some kind of crash of a process.
 Users reported either rebooting the server, different hardware or just trying
@@ -202,7 +203,7 @@ See the installation instructions for a full list of required packages.
 ### I forgot to delete the flatnodes file before starting an import.
 
 That's fine. For each import the flatnodes file get overwritten.
-See [https://help.openstreetmap.org/questions/52419/nominatim-flatnode-storage]()
+See [https://help.openstreetmap.org/questions/52419/nominatim-flatnode-storage](https://help.openstreetmap.org/questions/52419/nominatim-flatnode-storage)
 for more information.
 
 
@@ -211,11 +212,3 @@ for more information.
 ### Can I import negative OSM ids into Nominatim?
 
 See [this question of Stackoverflow](https://help.openstreetmap.org/questions/64662/nominatim-flatnode-with-negative-id).
-
-### Missing XML or text declaration
-
-The website might show: `XML Parsing Error: XML or text declaration not at start of entity Location.`
-
-Make sure there are no spaces at the beginning of your `settings/local.php` file.
-
-

docs/admin/Import.md

@@ -1,22 +1,54 @@
 # Importing the Database
 
 The following instructions explain how to create a Nominatim database
-from an OSM planet file and how to keep the database up to date. It
-is assumed that you have already successfully installed the Nominatim
-software itself, if not return to the [installation page](Installation.md).
+from an OSM planet file. It is assumed that you have already successfully
+installed the Nominatim software itself and the `nominatim` tool can be found
+in your `PATH`. If this is not the case, return to the
+[installation page](Installation.md).
 
-## Configuration setup in settings/local.php
+## Creating the project directory
 
-The Nominatim server can be customized via the file `settings/local.php`
-in the build directory. Note that this is a PHP file, so it must always
-start like this:
+Before you start the import, you should create a project directory for your
+new database installation. This directory receives all data that is related
+to a single Nominatim setup: configuration, extra data, etc. Create a project
+directory apart from the Nominatim software and change into the directory:
 
-    <?php
+```
+mkdir ~/nominatim-planet
+cd ~/nominatim-planet
+```
 
-without any leading spaces.
+In the following, we refer to the project directory as `$PROJECT_DIR`. To be
+able to copy&paste instructions, you can export the appropriate variable:
+
+```
+export PROJECT_DIR=~/nominatim-planet
+```
+
+The Nominatim tool assumes per default that the current working directory is
+the project directory but you may explicitly state a different directory using
+the `--project-dir` parameter. The following instructions assume that you run
+all commands from the project directory.
+
+!!! tip "Migration Tip"
+
+    Nominatim used to be run directly from the build directory until version 3.6.
+    Essentially, the build directory functioned as the project directory
+    for the database installation. This setup still works and can be useful for
+    development purposes. It is not recommended anymore for production setups.
+    Create a project directory that is separate from the Nominatim software.
+
+### Configuration setup in `.env`
+
+The Nominatim server can be customized via an `.env` configuration file in the
+project directory. This is a file in [dotenv](https://github.com/theskumar/python-dotenv)
+format which looks the same as variable settings in a standard shell environment.
+You can also set the same configuration via environment variables. All
+settings have a `NOMINATIM_` prefix to avoid conflicts with other environment
+variables.
 
 There are lots of configuration settings you can tweak. Have a look
-at `settings/default.php` for a full list. Most should have a sensible default.
+at `settings/env.default` for a full list. Most should have a sensible default.
 
 #### Flatnode files
 
@@ -24,9 +56,9 @@ If you plan to import a large dataset (e.g. Europe, North America, planet),
 you should also enable flatnode storage of node locations. With this
 setting enabled, node coordinates are stored in a simple file instead
 of the database. This will save you import time and disk storage.
-Add to your `settings/local.php`:
+Add to your `.env`:
 
-    @define('CONST_Osm2pgsql_Flatnode_File', '/path/to/flatnode.file');
+    NOMINATIM_FLATNODE_FILE="/path/to/flatnode.file"
 
 Replace the second part with a suitable path on your system and make sure
 the directory exists. There should be at least 75GB of free space.
@@ -38,9 +70,9 @@ the directory exists. There should be at least 75GB of free space.
 Wikipedia can be used as an optional auxiliary data source to help indicate
 the importance of OSM features. Nominatim will work without this information
 but it will improve the quality of the results if this is installed.
-This data is available as a binary download:
+This data is available as a binary download. Put it into your project directory:
 
-    cd $NOMINATIM_SOURCE_DIR/data
+    cd $PROJECT_DIR
     wget https://www.nominatim.org/data/wikimedia-importance.sql.gz
 
 The file is about 400MB and adds around 4GB to the Nominatim database.
@@ -48,15 +80,16 @@ The file is about 400MB and adds around 4GB to the Nominatim database.
 !!! tip
     If you forgot to download the wikipedia rankings, you can also add
     importances after the import. Download the files, then run
-    `./utils/setup.php --import-wikipedia-articles`
-    and `./utils/update.php --recompute-importance`.
+    `nominatim refresh --wiki-data --importance`. Updating importances for
+    a planet can take a couple of hours.
 
 ### Great Britain, USA postcodes
 
 Nominatim can use postcodes from an external source to improve searches that
-involve a GB or US postcode. This data can be optionally downloaded:
+involve a GB or US postcode. This data can be optionally downloaded into the
+project directory:
 
-    cd $NOMINATIM_SOURCE_DIR/data
+    cd $PROJECT_DIR
     wget https://www.nominatim.org/data/gb_postcode_data.sql.gz
     wget https://www.nominatim.org/data/us_postcode_data.sql.gz
 
@@ -86,11 +119,14 @@ that Nominatim cannot compute the areas for some administrative areas.
 About half of the data in Nominatim's database is not really used for serving
 the API. It is only there to allow the data to be updated from the latest
 changes from OSM. For many uses these dynamic updates are not really required.
-If you don't plan to apply updates, the dynamic part of the database can be
-safely dropped using the following command:
+If you don't plan to apply updates, you can run the import with the
+`--no-updates` parameter. This will drop the dynamic part of the database as
+soon as it is not required anymore.
+
+You can also drop the dynamic part later using the following command:
 
 ```
-./utils/setup.php --drop
+nominatim freeze
 ```
 
 Note that you still need to provide for sufficient disk space for the initial
@@ -124,7 +160,7 @@ import styles available which only read selected data:
   Like the full style but also adds most of the OSM tags into the extratags
   column.
 
-The style can be changed with the configuration `CONST_Import_Style`.
+The style can be changed with the configuration `NOMINATIM_IMPORT_STYLE`.
 
 To give you an idea of the impact of using the different styles, the table
 below gives rough estimates of the final database size after import of a
@@ -156,7 +192,7 @@ Download the data to import. Then issue the following command
 from the **build directory** to start the import:
 
 ```sh
-./utils/setup.php --osm-file <data file> --all 2>&1 | tee setup.log
+nominatim import --osm-file <data file> 2>&1 | tee setup.log
 ```
 
 ### Notes on full planet imports
@@ -192,29 +228,19 @@ MB. Make sure you leave enough RAM for PostgreSQL and osm2pgsql as mentioned
 above. If the system starts swapping or you are getting out-of-memory errors,
 reduce the cache size or even consider using a flatnode file.
 
-### Verify the import
+
+### Testing the installation
 
 Run this script to verify all required tables and indices got created successfully.
 
 ```sh
-./utils/check_import_finished.php
+nominatim admin --check-database
 ```
 
-### Setting up the website
-
-Run the following command to set up the configuration file for the API frontend
-`settings/settings-frontend.php`. These settings are used in website/*.php files.
-
-```sh
-./utils/setup.php --setup-website
-```
-!!! Note
-    This step is not necessary if you use `--all` option while setting up the DB.
-
 Now you can try out your installation by running:
 
 ```sh
-make serve
+nominatim serve
 ```
 
 This runs a small test server normally used for development. You can use it
@@ -232,7 +258,7 @@ planner to make the right decisions. Recomputing them can improve the performanc
 of forward geocoding in particular under high load. To recompute word counts run:
 
 ```sh
-./utils/update.php --recompute-word-counts
+nominatim refresh --word-counts
 ```
 
 This will take a couple of hours for a full planet installation. You can
@@ -242,10 +268,9 @@ running this function.
 
 If you want to be able to search for places by their type through
 [special key phrases](https://wiki.openstreetmap.org/wiki/Nominatim/Special_Phrases)
-you also need to enable these key phrases like this:
+you also need to import these key phrases like this:
 
-    ./utils/specialphrases.php --wiki-import > specialphrases.sql
-    psql -d nominatim -f specialphrases.sql
+    nominatim special-phrases --import-from-wiki
 
 Note that this command downloads the phrases from the wiki link above. You
 need internet access for the step.
@@ -258,26 +283,22 @@ address set to complement the OSM house number data in the US. You can add
 TIGER data to your own Nominatim instance by following these steps. The
 entire US adds about 10GB to your database.
 
-  1. Get preprocessed TIGER 2019 data and unpack it into the
-     data directory in your Nominatim sources:
+  1. Get preprocessed TIGER 2020 data:
 
-        cd Nominatim/data
-        wget https://nominatim.org/data/tiger2019-nominatim-preprocessed.tar.gz
-        tar xf tiger2019-nominatim-preprocessed.tar.gz
+        cd $PROJECT_DIR
+        wget https://nominatim.org/data/tiger2020-nominatim-preprocessed.tar.gz
 
   2. Import the data into your Nominatim database:
 
-        ./utils/setup.php --import-tiger-data
+        nominatim add-data --tiger-data tiger2020-nominatim-preprocessed.tar.gz
 
-  3. Enable use of the Tiger data in your `settings/local.php` by adding:
+  3. Enable use of the Tiger data in your `.env` by adding:
 
-         @define('CONST_Use_US_Tiger_Data', true);
+        echo NOMINATIM_USE_US_TIGER_DATA=yes >> .env
 
   4. Apply the new settings:
 
-```sh
-    ./utils/setup.php --create-functions --enable-diff-updates --create-partition-functions
-```
+        nominatim refresh --functions
 
 
 See the [developer's guide](../develop/data-sources.md#us-census-tiger) for more

docs/admin/Installation.md

@@ -30,23 +30,29 @@ For compiling:
   * [proj](https://proj.org/)
   * [bzip2](http://www.bzip.org/)
   * [zlib](https://www.zlib.net/)
+  * [ICU](http://site.icu-project.org/)
   * [Boost libraries](https://www.boost.org/), including system and filesystem
   * PostgreSQL client libraries
   * a recent C++ compiler (gcc 5+ or Clang 3.8+)
 
 For running Nominatim:
 
-  * [PostgreSQL](https://www.postgresql.org) (9.3+)
+  * [PostgreSQL](https://www.postgresql.org) (9.3+ will work, 11+ strongly recommended)
   * [PostGIS](https://postgis.net) (2.2+)
-  * [Python 3](https://www.python.org/)
-  * [Psycopg2](https://www.psycopg.org)
+  * [Python 3](https://www.python.org/) (3.5+)
+  * [Psycopg2](https://www.psycopg.org) (2.7+)
+  * [Python Dotenv](https://github.com/theskumar/python-dotenv)
+  * [psutil](https://github.com/giampaolo/psutil)
+  * [Jinja2](https://palletsprojects.com/p/jinja/)
+  * [PyICU](https://pypi.org/project/PyICU/)
   * [PHP](https://php.net) (7.0 or later)
   * PHP-pgsql
   * PHP-intl (bundled with PHP)
+  * PHP-cgi (for running queries from the command line)
 
 For running continuous updates:
 
-  * [pyosmium](https://osmcode.org/pyosmium/) (with Python 3)
+  * [pyosmium](https://osmcode.org/pyosmium/)
 
 For dependencies for running tests and building documentation, see
 the [Development section](../develop/Development-Environment.md).
@@ -142,6 +148,16 @@ build at the same level as the Nominatim source directory run:
 ```
 cmake ../Nominatim
 make
+sudo make install
+```
+
+Nominatim installs itself into `/usr/local` per default. To choose a different
+installation directory add `-DCMAKE_INSTALL_PREFIX=<install root>` to the
+cmake command. Make sure that the `bin` directory is available in your path
+in that case, e.g.
+
+```
+export PATH=<install root>/bin:$PATH
 ```
 
 Now continue with [importing the database](Import.md).

docs/admin/Migration.md

@@ -1,10 +1,55 @@
 # Database Migrations
 
-This page describes database migrations necessary to update existing databases
-to newer versions of Nominatim.
+Since version 3.7.0 Nominatim offers automatic migrations. Please follow
+the following steps:
 
-SQL statements should be executed from the PostgreSQL commandline. Execute
-`psql nominatim` to enter command line mode.
+* stop any updates that are potentially running
+* update Nominatim to the newer version
+* go to your project directory and run `nominatim admin --migrate`
+* (optionally) restart updates
+
+Below you find additional migrations and hints about other structural and
+breaking changes. **Please read them before running the migration.**
+
+!!! note
+    If you are migrating from a version <3.6, then you still have to follow
+    the manual migration steps up to 3.6.
+
+## 3.6.0 -> 3.7.0
+
+### New location for data files
+
+External data files for Wikipedia importance, postcodes etc. are no longer
+expected to reside in the source tree by default. Instead they will be searched
+in the project directory. If you have an automated setup script you must
+either adapt the download location or explicitly set the location of the
+files to the old place in your `.env`.
+
+### Introducing `nominatim` command line tool
+
+The various php utilities have been replaced with a single `nominatim`
+command line tool. Make sure to adapt any scripts. There is no direct 1:1
+matching between the old utilities and the commands of nominatim CLI. The
+following list gives you a list of nominatim sub-commands that contain
+functionality of each script:
+
+* ./utils/setup.php: `import`, `freeze`, `refresh`
+* ./utils/update.php: `replication`, `add-data`, `index`, `refresh`
+* ./utils/specialphrases.php: `special-phrases`
+* ./utils/check_import_finished.php: `admin`
+* ./utils/warm.php: `admin`
+* ./utils/export.php: `export`
+
+Try `nominatim <command> --help` for more information about each subcommand.
+
+`./utils/query.php` no longer exists in its old form. `nominatim search`
+provides a replacement but returns different output.
+
+### Switch to normalized house numbers
+
+The housenumber column in the placex table uses now normalized version.
+The automatic migration step will convert the column but this may take a
+very long time. It is advisable to take the machine offline while doing that.
 
 ## 3.5.0 -> 3.6.0
 

docs/admin/Update.md

@@ -1,8 +1,10 @@
 # Updating the Database
 
 There are many different ways to update your Nominatim database.
-The following section describes how to keep it up-to-date with Pyosmium.
-For a list of other methods see the output of `./utils/update.php --help`.
+The following section describes how to keep it up-to-date using
+an [online replication service for OpenStreetMap data](https://wiki.openstreetmap.org/wiki/Planet.osm/diffs)
+For a list of other methods to add or update data see the output of
+`nominatim add-data --help`.
 
 !!! important
     If you have configured a flatnode file for the import, then you
@@ -17,50 +19,37 @@ Run (as the same user who will later run the updates):
 pip3 install --user osmium
 ```
 
-Nominatim needs a tool called `pyosmium-get-changes` which comes with
-Pyosmium. You need to tell Nominatim where to find it. Add the
-following line to your `settings/local.php`:
-
-    @define('CONST_Pyosmium_Binary', '/home/user/.local/bin/pyosmium-get-changes');
-
-The path above is fine if you used the `--user` parameter with pip.
-Replace `user` with your user name.
-
 #### Setting up the update process
 
 Next the update needs to be initialised. By default Nominatim is configured
 to update using the global minutely diffs.
 
 If you want a different update source you will need to add some settings
-to `settings/local.php`. For example, to use the daily country extracts
+to `.env`. For example, to use the daily country extracts
 diffs for Ireland from Geofabrik add the following:
 
-    // base URL of the replication service
-    @define('CONST_Replication_Url', 'https://download.geofabrik.de/europe/ireland-and-northern-ireland-updates');
-    // How often upstream publishes diffs
-    @define('CONST_Replication_Update_Interval', '86400');
-    // How long to sleep if no update found yet
-    @define('CONST_Replication_Recheck_Interval', '900');
+    # base URL of the replication service
+    NOMINATIM_REPLICATION_URL="https://download.geofabrik.de/europe/ireland-and-northern-ireland-updates"
+    # How often upstream publishes diffs
+    NOMINATIM_REPLICATION_UPDATE_INTERVAL=86400
+    # How long to sleep if no update found yet
+    NOMINATIM_REPLICATION_RECHECK_INTERVAL=900
 
 To set up the update process now run the following command:
 
-    ./utils/update.php --init-updates
+    nominatim replication --init
 
 It outputs the date where updates will start. Recheck that this date is
 what you expect.
 
-The `--init-updates` command needs to be rerun whenever the replication service
-is changed.
+The `replication --init` command needs to be rerun whenever the replication
+service is changed.
 
 #### Updating Nominatim
 
 The following command will keep your database constantly up to date:
 
-    ./utils/update.php --import-osmosis-all
-
-(Note that even though the old name "import-osmosis-all" has been kept for
-compatibility reasons, Osmosis is not required to run this - it uses pyosmium
-behind the scenes.)
+    nominatim replication
 
 If you have imported multiple country extracts and want to keep them
 up-to-date, [Advanced installations section](Advanced-Installations.md) contains instructions 

docs/api/Faq.md

@@ -58,4 +58,4 @@ The [Overpass API](https://wiki.openstreetmap.org/wiki/Overpass_API) is more
 suited for these kinds of queries.
 
 That said if you installed your own Nominatim instance you can use the
-`/utils/export.php` PHP script as basis to return such lists.
+`nominatim export` PHP script as basis to return such lists.

docs/api/Reverse.md

@@ -162,7 +162,7 @@ This overrides the specified machine readable format. (Default: 0)
   "licence":"Data © OpenStreetMap contributors, ODbL 1.0. https:\/\/www.openstreetmap.org\/copyright",
   "osm_type":"way",
   "osm_id":"280940520",
-"lat":"-34.4391708",
+  "lat":"-34.4391708",
   "lon":"-58.7064573",
   "place_rank":"26",
   "category":"highway",

docs/api/Status.md

@@ -35,10 +35,16 @@ will return HTTP code 200 and a structure
   {
       "status": 0,
       "message": "OK",
-      "data_updated": "2020-05-04T14:47:00+00:00"
+      "data_updated": "2020-05-04T14:47:00+00:00",
+      "software_version": "3.6.0-0",
+      "database_version": "3.6.0-0"
   }
 ```
 
+The `software_version` field contains the version of Nominatim used to serve
+the API. The `database_version` field contains the version of the data format
+in the database.
+
 On error will also return HTTP status code 200 and a structure with error
 code and message, e.g.
 

docs/develop/Development-Environment.md

@@ -26,12 +26,14 @@ following packages should get you started:
 ## Prerequisites for testing and documentation
 
 The Nominatim test suite consists of behavioural tests (using behave) and
-unit tests (using PHPUnit). It has the following additional requirements:
+unit tests (using PHPUnit for PHP code and pytest for Python code).
+It has the following additional requirements:
 
 * [behave test framework](https://behave.readthedocs.io) >= 1.2.5
-* [nose](https://nose.readthedocs.io)
 * [phpunit](https://phpunit.de) >= 7.3
 * [PHP CodeSniffer](https://github.com/squizlabs/PHP_CodeSniffer)
+* [Pylint](https://pylint.org/) (2.6.0 is used for the CI)
+* [pytest](https://pytest.org)
 
 The documentation is built with mkdocs:
 
@@ -47,9 +49,9 @@ To install all necessary packages run:
 
 ```sh
 sudo apt install php-cgi phpunit php-codesniffer \
-                 python3-pip python3-setuptools python3-dev
+                 python3-pip python3-setuptools python3-dev pylint
 
-pip3 install --user behave nose mkdocs
+pip3 install --user behave mkdocs pytest
 ```
 
 The `mkdocs` executable will be located in `.local/bin`. You may have to add
@@ -78,58 +80,15 @@ echo 'export PATH=~/.config/composer/vendor/bin:$PATH' > ~/.profile
 
 ## Executing Tests
 
-All tests are located in the `\test` directory.
+All tests are located in the `/test` directory.
 
-### Preparing the test database
-
-Some of the behavioural test expect a test database to be present. You need at
-least 2GB RAM and 10GB disk space to create the database.
-
-First create a separate directory for the test DB and fetch the test planet
-data and the Tiger data for South Dakota:
-
-```
-mkdir testdb
-cd testdb
-wget https://www.nominatim.org/data/test/nominatim-api-testdata.pbf
-wget -O - https://nominatim.org/data/tiger2018-nominatim-preprocessed.tar.gz | tar xz --wildcards --no-anchored '46*'
-```
-
-Configure and build Nominatim in the usual way:
-
-```
-cmake $USERNAME/Nominatim
-make
-```
-
-Copy the test settings:
-
-```
-cp $USERNAME/Nominatim/test/testdb/local.php settings/
-```
-
-Inspect the file to check that all settings are correct for your local setup.
-
-Now you can import the test database:
-
-```
-dropdb --if-exists test_api_nominatim
-./utils/setup.php --all --osm-file nominatim-api-testdb.pbf 2>&1 | tee import.log
-./utils/specialphrases.php --wiki-import | psql -d test_api_nominatim 2>&1 | tee -a import.log
-./utils/setup.php --import-tiger-data 2>&1 | tee -a import.log
-```
-
-### Running the tests
-
-To run all tests just go to the test directory and run make:
+To run all tests just go to the build directory and run make:
 
 ```sh
-cd test
-make
+cd build
+make test
 ```
 
-To skip tests that require the test database, run `make no-test-db` instead.
-
 For more information about the structure of the tests and how to change and
 extend the test suite, see the [Testing chapter](Testing.md).
 

docs/develop/Import.md

@@ -29,7 +29,7 @@ once with `class` of `highway` and once with a `class` of `bridge`. Thus the
 ## Configuring the Import
 
 How tags are interpreted and assigned to the different `place` columns can be
-configured via the import style configuration file (`CONST_Import_style`). This
+configured via the import style configuration file (`NOMINATIM_IMPORT_STYLE`). This
 is a JSON file which contains a list of rules which are matched against every
 tag of every object and then assign the tag its specific role.
 

docs/develop/Postcodes.md

@@ -14,7 +14,7 @@ country's format, e.g. if Swiss postcodes are 4 digits.
 ## Regular updating calculated postcodes
 
 The script to rerun the calculation is
-`build/utils/update.php --calculate-postcodes`
+`nominatim refresh --postcodes`
 and runs once per night on nominatim.openstreetmap.org.
 
 

docs/develop/Ranking.md

@@ -87,9 +87,9 @@ into the database. There are a few hard-coded rules for the assignment:
     * highway nodes
     * landuse that is not an area
 
-Other than that, the ranks can be freely assigned via the JSON file
-defined with `CONST_Address_Level_Config` according to their type and
-the country they are in.
+Other than that, the ranks can be freely assigned via the JSON file according
+to their type and the country they are in. The name of the config file to be
+used can be changed with the setting `NOMINATIM_ADDRESS_LEVEL_CONFIG`.
 
 The address level configuration must consist of an array of configuration
 entries, each containing a tag definition and an optional country array:

docs/develop/Testing.md

@@ -21,14 +21,15 @@ This test directory is sturctured as follows:
   | +-  api        Tests for API endpoints (search, reverse, etc.)
   |
   +-   php         PHP unit tests
+  +-   python      Python unit tests
   +-   scenes      Geometry test data
   +-   testdb      Base data for generating API test database
 ```
 
 ## PHP Unit Tests (`test/php`)
 
-Unit tests can be found in the php/ directory. They test selected php functions.
-Very low coverage.
+Unit tests for PHP code can be found in the `php/` directory. They test selected
+PHP functions. Very low coverage.
 
 To execute the test suite run
 
@@ -36,11 +37,26 @@ To execute the test suite run
     UNIT_TEST_DSN='pgsql:dbname=nominatim_unit_tests' phpunit ../
 
 It will read phpunit.xml which points to the library, test path, bootstrap
-strip and set other parameters.
+strip and sets other parameters.
 
 It will use (and destroy) a local database 'nominatim_unit_tests'. You can set
 a different connection string with e.g. UNIT_TEST_DSN='pgsql:dbname=foo_unit_tests'.
 
+## Python Unit Tests (`test/python`)
+
+Unit tests for Python code can be found in the `python/` directory. The goal is
+to have complete coverage of the Python library in `nominatim`.
+
+To execute the tests run
+
+    py.test-3 test/python
+
+or
+
+    pytest test/python
+
+The name of the pytest binary depends on your installation.
+
 ## BDD Functional Tests (`test/bdd`)
 
 Functional tests are written as BDD instructions. For more information on
@@ -67,17 +83,17 @@ The tests can be configured with a set of environment variables (`behave -D key=
                    the test databases (db tests)
  * `TEST_DB` - name of test database (db tests)
  * `API_TEST_DB` - name of the database containing the API test data (api tests)
+ * `API_TEST_FILE` - OSM file to be imported into the API test database (api tests)
  * `DB_HOST` - (optional) hostname of database host
  * `DB_PORT` - (optional) port of database on host
  * `DB_USER` - (optional) username of database login
  * `DB_PASS` - (optional) password for database login
  * `SERVER_MODULE_PATH` - (optional) path on the Postgres server to Nominatim
                           module shared library file
- * `TEST_SETTINGS_TEMPLATE` - file to write temporary Nominatim settings to
- * `REMOVE_TEMPLATE` - if true, the template database will not be reused during
-                       the next run. Reusing the base templates speeds up tests
-                       considerably but might lead to outdated errors for some
-                       changes in the database layout.
+ * `REMOVE_TEMPLATE` - if true, the template and API database will not be reused
+                       during the next run. Reusing the base templates speeds
+                       up tests considerably but might lead to outdated errors
+                       for some changes in the database layout.
  * `KEEP_TEST_DB` - if true, the test database will not be dropped after a test
                     is finished. Should only be used if one single scenario is
                     run, otherwise the result is undefined.
@@ -89,23 +105,20 @@ feature of behave which comes in handy when writing new tests.
 ### API Tests (`test/bdd/api`)
 
 These tests are meant to test the different API endpoints and their parameters.
-They require to import several datasets into a test database.
-See the [Development Setup chapter](Development-Environment.md#preparing-the-test-database)
-for instructions on how to set up this database.
+They require to import several datasets into a test database. This is normally
+done automatically during setup of the test. The API test database is then
+kept around and reused in subsequent runs of behave. Use `behave -DREMOVE_TEMPLATE`
+to force a reimport of the database.
 
-The official test dataset was derived from the 180924 planet (note: such
-file no longer exists at https://planet.openstreetmap.org/planet/2018/).
-Newer planets are likely to work as well but you may see isolated test
-failures where the data has changed.
+The official test dataset is saved in the file `test/testdb/apidb-test-data.pbf`
+and compromises the following data:
 
-The official test dataset can always be downloaded from
-[nominatim.org](https://www.nominatim.org/data/test/nominatim-api-testdata.pbf)
-To recreate the input data for the test database run:
+ * Geofabrik extract of Liechtenstein
+ * extract of Autauga country, Alabama, US (for tests against Tiger data)
+ * additional data from `test/testdb/additional_api_test.data.osm`
 
-```
-wget https://ftp5.gwdg.de/pub/misc/openstreetmap/planet.openstreetmap.org/pbf/planet-180924.osm.pbf
-osmconvert planet-180924.osm.pbf -B=test/testdb/testdb.polys -o=testdb.pbf
-```
+API tests should only be testing the functionality of the website PHP code.
+Most tests should be formulated as BDD DB creation tests (see below) instead.
 
 #### Code Coverage
 
@@ -140,3 +153,7 @@ needs superuser rights for postgres.
 
 These tests check that data is imported correctly into the place table. They
 use the same template database as the DB Creation tests, so the same remarks apply.
+
+Note that most testing of the gazetteer output of osm2pgsql is done in the tests
+of osm2pgsql itself. The BDD tests are just there to ensure compatibility of
+the osm2pgsql and Nominatim code.

docs/mkdocs.yml

@@ -1,4 +1,4 @@
-site_name: Nominatim 3.6.0
+site_name: Nominatim Documentation
 theme: readthedocs
 docs_dir: ${CMAKE_CURRENT_BINARY_DIR}
 site_url: https://nominatim.org

lib-php/AddressDetails.php

@@ -2,7 +2,7 @@
 
 namespace Nominatim;
 
-require_once(CONST_BasePath.'/lib/ClassTypes.php');
+require_once(CONST_LibDir.'/ClassTypes.php');
 
 /**
  * Detailed list of address parts for a single result

lib-php/DB.php

@@ -2,7 +2,7 @@
 
 namespace Nominatim;
 
-require_once(CONST_BasePath.'/lib/DatabaseError.php');
+require_once(CONST_LibDir.'/DatabaseError.php');
 
 /**
  * Uses PDO to access the database specified in the CONST_Database_DSN
@@ -12,9 +12,9 @@ class DB
 {
     protected $connection;
 
-    public function __construct($sDSN = CONST_Database_DSN)
+    public function __construct($sDSN = null)
     {
-        $this->sDSN = $sDSN;
+        $this->sDSN = $sDSN ?? getSetting('DATABASE_DSN');
     }
 
     public function connect($bNew = false, $bPersistent = true)
@@ -240,16 +240,6 @@ class DB
         return ($this->getOne($sSQL, array(':tablename' => $sTableName)) == 1);
     }
 
-    /**
-    * Returns a list of table names in the database
-    *
-    * @return array[]
-    */
-    public function getListOfTables()
-    {
-        return $this->getCol("SELECT tablename FROM pg_tables WHERE schemaname='public'");
-    }
-
     /**
      * Deletes a table. Returns true if deleted or didn't exist.
      *
@@ -262,76 +252,6 @@ class DB
         return $this->exec('DROP TABLE IF EXISTS '.$sTableName.' CASCADE') == 0;
     }
 
-    /**
-    * Check if an index exists in the database. Optional filtered by tablename
-    *
-    * @param string  $sTableName
-    *
-    * @return boolean
-    */
-    public function indexExists($sIndexName, $sTableName = null)
-    {
-        return in_array($sIndexName, $this->getListOfIndices($sTableName));
-    }
-
-    /**
-    * Returns a list of index names in the database, optional filtered by tablename
-    *
-    * @param string  $sTableName
-    *
-    * @return array
-    */
-    public function getListOfIndices($sTableName = null)
-    {
-        //  table_name            | index_name                      | column_name
-        // -----------------------+---------------------------------+--------------
-        //  country_name          | idx_country_name_country_code   | country_code
-        //  country_osm_grid      | idx_country_osm_grid_geometry   | geometry
-        //  import_polygon_delete | idx_import_polygon_delete_osmid | osm_id
-        //  import_polygon_delete | idx_import_polygon_delete_osmid | osm_type
-        //  import_polygon_error  | idx_import_polygon_error_osmid  | osm_id
-        //  import_polygon_error  | idx_import_polygon_error_osmid  | osm_type
-        $sSql = <<< END
-SELECT
-    t.relname as table_name,
-    i.relname as index_name,
-    a.attname as column_name
-FROM
-    pg_class t,
-    pg_class i,
-    pg_index ix,
-    pg_attribute a
-WHERE
-    t.oid = ix.indrelid
-    and i.oid = ix.indexrelid
-    and a.attrelid = t.oid
-    and a.attnum = ANY(ix.indkey)
-    and t.relkind = 'r'
-    and i.relname NOT LIKE 'pg_%'
-    FILTERS
- ORDER BY
-    t.relname,
-    i.relname,
-    a.attname
-END;
-
-        $aRows = null;
-        if ($sTableName) {
-            $sSql = str_replace('FILTERS', 'and t.relname = :tablename', $sSql);
-            $aRows = $this->getAll($sSql, array(':tablename' => $sTableName));
-        } else {
-            $sSql = str_replace('FILTERS', '', $sSql);
-            $aRows = $this->getAll($sSql);
-        }
-
-        $aIndexNames = array_unique(array_map(function ($aRow) {
-            return $aRow['index_name'];
-        }, $aRows));
-        sort($aIndexNames);
-
-        return $aIndexNames;
-    }
-
     /**
      * Tries to connect to the database but on failure doesn't throw an exception.
      *

lib-php/Geocode.php

@@ -2,12 +2,12 @@
 
 namespace Nominatim;
 
-require_once(CONST_BasePath.'/lib/PlaceLookup.php');
-require_once(CONST_BasePath.'/lib/Phrase.php');
-require_once(CONST_BasePath.'/lib/ReverseGeocode.php');
-require_once(CONST_BasePath.'/lib/SearchDescription.php');
-require_once(CONST_BasePath.'/lib/SearchContext.php');
-require_once(CONST_BasePath.'/lib/TokenList.php');
+require_once(CONST_LibDir.'/PlaceLookup.php');
+require_once(CONST_LibDir.'/Phrase.php');
+require_once(CONST_LibDir.'/ReverseGeocode.php');
+require_once(CONST_LibDir.'/SearchDescription.php');
+require_once(CONST_LibDir.'/SearchContext.php');
+require_once(CONST_LibDir.'/TokenList.php');
 
 class Geocode
 {
@@ -18,7 +18,7 @@ class Geocode
     protected $aLangPrefOrder = array();
 
     protected $aExcludePlaceIDs = array();
-    protected $bReverseInPlan = false;
+    protected $bReverseInPlan = true;
 
     protected $iLimit = 20;
     protected $iFinalLimit = 10;
@@ -778,14 +778,19 @@ class Geocode
                 if (!empty($aResults)) {
                     $aSplitResults = Result::splitResults($aResults);
                     Debug::printVar('Split results', $aSplitResults);
-                    if ($iGroupLoop <= 4 && empty($aSplitResults['tail'])
-                        && reset($aSplitResults['head'])->iResultRank > 0) {
+                    if ($iGroupLoop <= 4
+                        && reset($aSplitResults['head'])->iResultRank > 0
+                        && $iGroupedRank !== array_key_last($aGroupedSearches)) {
                         // Haven't found an exact match for the query yet.
                         // Therefore add result from the next group level.
                         $aNextResults = $aSplitResults['head'];
                         foreach ($aNextResults as $oRes) {
                             $oRes->iResultRank--;
                         }
+                        foreach ($aSplitResults['tail'] as $oRes) {
+                            $oRes->iResultRank--;
+                            $aNextResults[$oRes->iId] = $oRes;
+                        }
                         $aResults = array();
                     } else {
                         $aResults = $aSplitResults['head'];

lib-php/PlaceLookup.php

@@ -2,8 +2,8 @@
 
 namespace Nominatim;
 
-require_once(CONST_BasePath.'/lib/AddressDetails.php');
-require_once(CONST_BasePath.'/lib/Result.php');
+require_once(CONST_LibDir.'/AddressDetails.php');
+require_once(CONST_LibDir.'/Result.php');
 
 class PlaceLookup
 {
@@ -486,65 +486,63 @@ class PlaceLookup
         $aOutlineResult = array();
         if (!$iPlaceID) return $aOutlineResult;
 
-        if (CONST_Search_AreaPolygons) {
-            // Get the bounding box and outline polygon
-            $sSQL = 'select place_id,0 as numfeatures,st_area(geometry) as area,';
-            if ($fLonReverse != null && $fLatReverse != null) {
-                $sSQL .= ' ST_Y(closest_point) as centrelat,';
-                $sSQL .= ' ST_X(closest_point) as centrelon,';
-            } else {
-                $sSQL .= ' ST_Y(centroid) as centrelat, ST_X(centroid) as centrelon,';
-            }
-            $sSQL .= ' ST_YMin(geometry) as minlat,ST_YMax(geometry) as maxlat,';
-            $sSQL .= ' ST_XMin(geometry) as minlon,ST_XMax(geometry) as maxlon';
-            if ($this->bIncludePolygonAsGeoJSON) $sSQL .= ',ST_AsGeoJSON(geometry) as asgeojson';
-            if ($this->bIncludePolygonAsKML) $sSQL .= ',ST_AsKML(geometry) as askml';
-            if ($this->bIncludePolygonAsSVG) $sSQL .= ',ST_AsSVG(geometry) as assvg';
-            if ($this->bIncludePolygonAsText) $sSQL .= ',ST_AsText(geometry) as astext';
-            if ($fLonReverse != null && $fLatReverse != null) {
-                $sFrom = ' from (SELECT * , CASE WHEN (class = \'highway\') AND (ST_GeometryType(geometry) = \'ST_LineString\') THEN ';
-                $sFrom .=' ST_ClosestPoint(geometry, ST_SetSRID(ST_Point('.$fLatReverse.','.$fLonReverse.'),4326))';
-                $sFrom .=' ELSE centroid END AS closest_point';
-                $sFrom .= ' from placex where place_id = '.$iPlaceID.') as plx';
-            } else {
-                $sFrom = ' from placex where place_id = '.$iPlaceID;
-            }
-            if ($this->fPolygonSimplificationThreshold > 0) {
-                $sSQL .= ' from (select place_id,centroid,ST_SimplifyPreserveTopology(geometry,'.$this->fPolygonSimplificationThreshold.') as geometry'.$sFrom.') as plx';
-            } else {
-                $sSQL .= $sFrom;
+        // Get the bounding box and outline polygon
+        $sSQL = 'select place_id,0 as numfeatures,st_area(geometry) as area,';
+        if ($fLonReverse != null && $fLatReverse != null) {
+            $sSQL .= ' ST_Y(closest_point) as centrelat,';
+            $sSQL .= ' ST_X(closest_point) as centrelon,';
+        } else {
+            $sSQL .= ' ST_Y(centroid) as centrelat, ST_X(centroid) as centrelon,';
+        }
+        $sSQL .= ' ST_YMin(geometry) as minlat,ST_YMax(geometry) as maxlat,';
+        $sSQL .= ' ST_XMin(geometry) as minlon,ST_XMax(geometry) as maxlon';
+        if ($this->bIncludePolygonAsGeoJSON) $sSQL .= ',ST_AsGeoJSON(geometry) as asgeojson';
+        if ($this->bIncludePolygonAsKML) $sSQL .= ',ST_AsKML(geometry) as askml';
+        if ($this->bIncludePolygonAsSVG) $sSQL .= ',ST_AsSVG(geometry) as assvg';
+        if ($this->bIncludePolygonAsText) $sSQL .= ',ST_AsText(geometry) as astext';
+        if ($fLonReverse != null && $fLatReverse != null) {
+            $sFrom = ' from (SELECT * , CASE WHEN (class = \'highway\') AND (ST_GeometryType(geometry) = \'ST_LineString\') THEN ';
+            $sFrom .=' ST_ClosestPoint(geometry, ST_SetSRID(ST_Point('.$fLatReverse.','.$fLonReverse.'),4326))';
+            $sFrom .=' ELSE centroid END AS closest_point';
+            $sFrom .= ' from placex where place_id = '.$iPlaceID.') as plx';
+        } else {
+            $sFrom = ' from placex where place_id = '.$iPlaceID;
+        }
+        if ($this->fPolygonSimplificationThreshold > 0) {
+            $sSQL .= ' from (select place_id,centroid,ST_SimplifyPreserveTopology(geometry,'.$this->fPolygonSimplificationThreshold.') as geometry'.$sFrom.') as plx';
+        } else {
+            $sSQL .= $sFrom;
+        }
+
+        $aPointPolygon = $this->oDB->getRow($sSQL, null, 'Could not get outline');
+
+        if ($aPointPolygon && $aPointPolygon['place_id']) {
+            if ($aPointPolygon['centrelon'] !== null && $aPointPolygon['centrelat'] !== null) {
+                $aOutlineResult['lat'] = $aPointPolygon['centrelat'];
+                $aOutlineResult['lon'] = $aPointPolygon['centrelon'];
             }
 
-            $aPointPolygon = $this->oDB->getRow($sSQL, null, 'Could not get outline');
+            if ($this->bIncludePolygonAsGeoJSON) $aOutlineResult['asgeojson'] = $aPointPolygon['asgeojson'];
+            if ($this->bIncludePolygonAsKML) $aOutlineResult['askml'] = $aPointPolygon['askml'];
+            if ($this->bIncludePolygonAsSVG) $aOutlineResult['assvg'] = $aPointPolygon['assvg'];
+            if ($this->bIncludePolygonAsText) $aOutlineResult['astext'] = $aPointPolygon['astext'];
 
-            if ($aPointPolygon && $aPointPolygon['place_id']) {
-                if ($aPointPolygon['centrelon'] !== null && $aPointPolygon['centrelat'] !== null) {
-                    $aOutlineResult['lat'] = $aPointPolygon['centrelat'];
-                    $aOutlineResult['lon'] = $aPointPolygon['centrelon'];
-                }
-
-                if ($this->bIncludePolygonAsGeoJSON) $aOutlineResult['asgeojson'] = $aPointPolygon['asgeojson'];
-                if ($this->bIncludePolygonAsKML) $aOutlineResult['askml'] = $aPointPolygon['askml'];
-                if ($this->bIncludePolygonAsSVG) $aOutlineResult['assvg'] = $aPointPolygon['assvg'];
-                if ($this->bIncludePolygonAsText) $aOutlineResult['astext'] = $aPointPolygon['astext'];
-
-                if (abs($aPointPolygon['minlat'] - $aPointPolygon['maxlat']) < 0.0000001) {
-                    $aPointPolygon['minlat'] = $aPointPolygon['minlat'] - $fRadius;
-                    $aPointPolygon['maxlat'] = $aPointPolygon['maxlat'] + $fRadius;
-                }
-
-                if (abs($aPointPolygon['minlon'] - $aPointPolygon['maxlon']) < 0.0000001) {
-                    $aPointPolygon['minlon'] = $aPointPolygon['minlon'] - $fRadius;
-                    $aPointPolygon['maxlon'] = $aPointPolygon['maxlon'] + $fRadius;
-                }
-
-                $aOutlineResult['aBoundingBox'] = array(
-                                                   (string)$aPointPolygon['minlat'],
-                                                   (string)$aPointPolygon['maxlat'],
-                                                   (string)$aPointPolygon['minlon'],
-                                                   (string)$aPointPolygon['maxlon']
-                                                  );
+            if (abs($aPointPolygon['minlat'] - $aPointPolygon['maxlat']) < 0.0000001) {
+                $aPointPolygon['minlat'] = $aPointPolygon['minlat'] - $fRadius;
+                $aPointPolygon['maxlat'] = $aPointPolygon['maxlat'] + $fRadius;
             }
+
+            if (abs($aPointPolygon['minlon'] - $aPointPolygon['maxlon']) < 0.0000001) {
+                $aPointPolygon['minlon'] = $aPointPolygon['minlon'] - $fRadius;
+                $aPointPolygon['maxlon'] = $aPointPolygon['maxlon'] + $fRadius;
+            }
+
+            $aOutlineResult['aBoundingBox'] = array(
+                                               (string)$aPointPolygon['minlat'],
+                                               (string)$aPointPolygon['maxlat'],
+                                               (string)$aPointPolygon['minlon'],
+                                               (string)$aPointPolygon['maxlon']
+                                              );
         }
 
         // as a fallback we generate a bounding box without knowing the size of the geometry

lib-php/Result.php

@@ -26,6 +26,8 @@ class Result
     public $iExactMatches = 0;
     /// Subranking within the results (the higher the worse).
     public $iResultRank = 0;
+    /// Address rank of the result.
+    public $iAddressRank;
 
     public function debugInfo()
     {
@@ -84,7 +86,7 @@ class Result
 
         foreach ($aResults as $oRes) {
             if ($oRes->iResultRank < $iMinRank) {
-                $aTail = array_merge($aTail, $aHead);
+                $aTail += $aHead;
                 $aHead = array($oRes->iId => $oRes);
                 $iMinRank = $oRes->iResultRank;
             } elseif ($oRes->iResultRank == $iMinRank) {

lib-php/ReverseGeocode.php

@@ -2,7 +2,7 @@
 
 namespace Nominatim;
 
-require_once(CONST_BasePath.'/lib/Result.php');
+require_once(CONST_LibDir.'/Result.php');
 
 class ReverseGeocode
 {
@@ -280,29 +280,6 @@ class ReverseGeocode
                 $iPlaceID = $aPlace['place_id'];
                 $oResult = new Result($iPlaceID);
                 $iRankAddress = $aPlace['rank_address'];
-                $iParentPlaceID = $aPlace['parent_place_id'];
-            }
-
-            if ($bDoInterpolation && $iMaxRank >= 30) {
-                $fDistance = $fSearchDiam;
-                if ($aPlace) {
-                    // We can't reliably go from the closest street to an
-                    // interpolation line because the closest interpolation
-                    // may have a different street segments as a parent.
-                    // Therefore allow an interpolation line to take precendence
-                    // even when the street is closer.
-                    $fDistance = $iRankAddress < 28 ? 0.001 : $aPlace['distance'];
-                }
-
-                $aHouse = $this->lookupInterpolation($sPointSQL, $fDistance);
-                Debug::printVar('Interpolation result', $aPlace);
-
-                if ($aHouse) {
-                    $oResult = new Result($aHouse['place_id'], Result::TABLE_OSMLINE);
-                    $oResult->iHouseNumber = closestHouseNumber($aHouse);
-                    $aPlace = $aHouse;
-                    $iRankAddress = 30;
-                }
             }
 
             if ($aPlace) {
@@ -328,7 +305,9 @@ class ReverseGeocode
                     Debug::printVar('Closest POI result', $aStreet);
 
                     if ($aStreet) {
+                        $aPlace = $aStreet;
                         $oResult = new Result($aStreet['place_id']);
+                        $iRankAddress = 30;
                     }
                 }
 
@@ -351,11 +330,37 @@ class ReverseGeocode
                     Debug::printVar('Tiger house number result', $aPlaceTiger);
 
                     if ($aPlaceTiger) {
+                        $aPlace = $aPlaceTiger;
                         $oResult = new Result($aPlaceTiger['place_id'], Result::TABLE_TIGER);
                         $oResult->iHouseNumber = closestHouseNumber($aPlaceTiger);
+                        $iRankAddress = 30;
                     }
                 }
-            } else {
+            }
+
+            if ($bDoInterpolation && $iMaxRank >= 30) {
+                $fDistance = $fSearchDiam;
+                if ($aPlace) {
+                    // We can't reliably go from the closest street to an
+                    // interpolation line because the closest interpolation
+                    // may have a different street segments as a parent.
+                    // Therefore allow an interpolation line to take precendence
+                    // even when the street is closer.
+                    $fDistance = $iRankAddress < 28 ? 0.001 : $aPlace['distance'];
+                }
+
+                $aHouse = $this->lookupInterpolation($sPointSQL, $fDistance);
+                Debug::printVar('Interpolation result', $aPlace);
+
+                if ($aHouse) {
+                    $oResult = new Result($aHouse['place_id'], Result::TABLE_OSMLINE);
+                    $oResult->iHouseNumber = closestHouseNumber($aHouse);
+                    $aPlace = $aHouse;
+                    $iRankAddress = 30;
+                }
+            }
+
+            if (!$aPlace) {
                 // if no POI or street is found ...
                 $oResult = $this->lookupLargeArea($sPointSQL, 25);
             }

lib-php/SearchContext.php

@@ -2,7 +2,7 @@
 
 namespace Nominatim;
 
-require_once(CONST_BasePath.'/lib/lib.php');
+require_once(CONST_LibDir.'/lib.php');
 
 
 /**

lib-php/SearchDescription.php

@@ -2,9 +2,9 @@
 
 namespace Nominatim;
 
-require_once(CONST_BasePath.'/lib/SpecialSearchOperator.php');
-require_once(CONST_BasePath.'/lib/SearchContext.php');
-require_once(CONST_BasePath.'/lib/Result.php');
+require_once(CONST_LibDir.'/SpecialSearchOperator.php');
+require_once(CONST_LibDir.'/SearchContext.php');
+require_once(CONST_LibDir.'/Result.php');
 
 /**
  * Description of a single interpretation of a search query.
@@ -86,18 +86,6 @@ class SearchDescription
         $this->sType = $sType;
     }
 
-    /**
-     * Check if this might be a full address search.
-     *
-     * @return bool True if the search contains name, address and housenumber.
-     */
-    public function looksLikeFullAddress()
-    {
-        return (!empty($this->aName))
-               && (!empty($this->aAddress) || $this->sCountryCode)
-               && preg_match('/[0-9]+/', $this->sHouseNumber);
-    }
-
     /**
      * Check if any operator is set.
      *
@@ -191,6 +179,7 @@ class SearchDescription
                 // - increase score for finding it anywhere else (optimisation)
                 if (!$bLastToken) {
                     $oSearch->iSearchRank += 5;
+                    $oSearch->iNamePhrase = -1;
                 }
                 $aNewSearches[] = $oSearch;
             }
@@ -217,6 +206,7 @@ class SearchDescription
                 ) {
                     $oSearch = clone $this;
                     $oSearch->iSearchRank++;
+                    $oSearch->iNamePhrase = -1;
                     if (strlen($oSearchTerm->sPostcode) < 4) {
                         $oSearch->iSearchRank += 4 - strlen($oSearchTerm->sPostcode);
                     }
@@ -230,7 +220,11 @@ class SearchDescription
             if (!$this->sHouseNumber && $this->iOperator != Operator::POSTCODE) {
                 $oSearch = clone $this;
                 $oSearch->iSearchRank++;
+                $oSearch->iNamePhrase = -1;
                 $oSearch->sHouseNumber = $oSearchTerm->sToken;
+                if ($this->iOperator != Operator::NONE) {
+                    $oSearch->iSearchRank++;
+                }
                 // sanity check: if the housenumber is not mainly made
                 // up of numbers, add a penalty
                 if (preg_match('/\\d/', $oSearch->sHouseNumber) === 0
@@ -267,7 +261,8 @@ class SearchDescription
         ) {
             if ($this->iOperator == Operator::NONE) {
                 $oSearch = clone $this;
-                $oSearch->iSearchRank++;
+                $oSearch->iSearchRank += 2;
+                $oSearch->iNamePhrase = -1;
 
                 $iOp = $oSearchTerm->iOperator;
                 if ($iOp == Operator::NONE) {
@@ -277,6 +272,11 @@ class SearchDescription
                         $iOp = Operator::NEAR;
                     }
                     $oSearch->iSearchRank += 2;
+                } elseif (!$bFirstToken && !$bLastToken) {
+                    $oSearch->iSearchRank += 2;
+                }
+                if ($this->sHouseNumber) {
+                    $oSearch->iSearchRank++;
                 }
 
                 $oSearch->setPoiSearch(
@@ -297,11 +297,12 @@ class SearchDescription
             if (!empty($this->aName) || !($bFirstPhrase || $sPhraseType == '')) {
                 if (($sPhraseType == '' || !$bFirstPhrase) && !$bHasPartial) {
                     $oSearch = clone $this;
+                    $oSearch->iNamePhrase = -1;
                     $oSearch->iSearchRank += 3 * $oSearchTerm->iTermCount;
                     $oSearch->aAddress[$iWordID] = $iWordID;
                     $aNewSearches[] = $oSearch;
                 }
-            } else {
+            } elseif (empty($this->aNameNonSearch)) {
                 $oSearch = clone $this;
                 $oSearch->iSearchRank++;
                 $oSearch->aName = array($iWordID => $iWordID);
@@ -341,51 +342,34 @@ class SearchDescription
 
         if ((!$bStructuredPhrases || $iPhrase > 0)
             && (!empty($this->aName))
-            && strpos($sToken, ' ') === false
         ) {
+            $oSearch = clone $this;
+            $oSearch->iSearchRank++;
+            if (preg_match('#^[0-9 ]+$#', $sToken)) {
+                $oSearch->iSearchRank++;
+            }
             if ($oSearchTerm->iSearchNameCount < CONST_Max_Word_Frequency) {
-                $oSearch = clone $this;
-                $oSearch->iSearchRank += $oSearchTerm->iTermCount + 1;
-                if (empty($this->aName)) {
-                    $oSearch->iSearchRank++;
-                }
-                if (preg_match('#^[0-9]+$#', $sToken)) {
-                    $oSearch->iSearchRank++;
-                }
                 $oSearch->aAddress[$iWordID] = $iWordID;
-                $aNewSearches[] = $oSearch;
             } else {
-                $oSearch = clone $this;
-                $oSearch->iSearchRank += $oSearchTerm->iTermCount + 1;
                 $oSearch->aAddressNonSearch[$iWordID] = $iWordID;
                 if (!empty($aFullTokens)) {
                     $oSearch->iSearchRank++;
                 }
-                $aNewSearches[] = $oSearch;
-
-                // revert to the token version?
-                foreach ($aFullTokens as $oSearchTermToken) {
-                    if (is_a($oSearchTermToken, '\Nominatim\Token\Word')) {
-                        $oSearch = clone $this;
-                        $oSearch->iSearchRank += 3;
-                        $oSearch->aAddress[$oSearchTermToken->iId]
-                            = $oSearchTermToken->iId;
-                        $aNewSearches[] = $oSearch;
-                    }
-                }
             }
+            $aNewSearches[] = $oSearch;
         }
 
         if ((!$this->sPostcode && !$this->aAddress && !$this->aAddressNonSearch)
-            && (empty($this->aName) || $this->iNamePhrase == $iPhrase)
+            && ((empty($this->aName) && empty($this->aNameNonSearch)) || $this->iNamePhrase == $iPhrase)
+            && strpos($sToken, ' ') === false
         ) {
             $oSearch = clone $this;
-            $oSearch->iSearchRank += 2;
-            if (empty($this->aName)) {
-                $oSearch->iSearchRank += 1;
+            $oSearch->iSearchRank++;
+            if (empty($this->aName) && empty($this->aNameNonSearch)) {
+                $oSearch->iSearchRank++;
             }
-            if (preg_match('#^[0-9]+$#', $sToken)) {
-                $oSearch->iSearchRank += 2;
+            if (preg_match('#^[0-9 ]+$#', $sToken)) {
+                $oSearch->iSearchRank++;
             }
             if ($oSearchTerm->iSearchNameCount < CONST_Max_Word_Frequency) {
                 if (empty($this->aName)
@@ -399,6 +383,9 @@ class SearchDescription
                 }
                 $oSearch->aName[$iWordID] = $iWordID;
             } else {
+                if (!empty($aFullTokens)) {
+                    $oSearch->iSearchRank++;
+                }
                 $oSearch->aNameNonSearch[$iWordID] = $iWordID;
             }
             $oSearch->iNamePhrase = $iPhrase;
@@ -462,7 +449,11 @@ class SearchDescription
                 // Downgrade the rank of the street results, they are missing
                 // the housenumber.
                 foreach ($aResults as $oRes) {
-                    $oRes->iResultRank++;
+                    if ($oRes->iAddressRank >= 26) {
+                        $oRes->iResultRank++;
+                    } else {
+                        $oRes->iResultRank += 2;
+                    }
                 }
 
                 $aHnResults = $this->queryHouseNumber($oDB, $aResults);
@@ -623,14 +614,14 @@ class SearchDescription
         // too many results are expected for the street, i.e. if the result
         // will be narrowed down by an address. Remeber that with ordering
         // every single result has to be checked.
-        if ($this->sHouseNumber && (!empty($this->aAddress) || $this->sPostcode)) {
+        if ($this->sHouseNumber && ($this->bRareName || !empty($this->aAddress) || $this->sPostcode)) {
             $sHouseNumberRegex = '\\\\m'.$this->sHouseNumber.'\\\\M';
             $aOrder[] = ' (';
             $aOrder[0] .= 'EXISTS(';
             $aOrder[0] .= '  SELECT place_id';
             $aOrder[0] .= '  FROM placex';
             $aOrder[0] .= '  WHERE parent_place_id = search_name.place_id';
-            $aOrder[0] .= "    AND transliteration(housenumber) ~* E'".$sHouseNumberRegex."'";
+            $aOrder[0] .= "    AND housenumber ~* E'".$sHouseNumberRegex."'";
             $aOrder[0] .= '  LIMIT 1';
             $aOrder[0] .= ') ';
             // also housenumbers from interpolation lines table are needed
@@ -727,7 +718,7 @@ class SearchDescription
         $aResults = array();
 
         if (!empty($aTerms)) {
-            $sSQL = 'SELECT place_id,'.$sExactMatchSQL;
+            $sSQL = 'SELECT place_id, address_rank,'.$sExactMatchSQL;
             $sSQL .= ' FROM search_name';
             $sSQL .= ' WHERE '.join(' and ', $aTerms);
             $sSQL .= ' ORDER BY '.join(', ', $aOrder);
@@ -740,6 +731,7 @@ class SearchDescription
             foreach ($aDBResults as $aResult) {
                 $oResult = new Result($aResult['place_id']);
                 $oResult->iExactMatches = $aResult['exactmatch'];
+                $oResult->iAddressRank = $aResult['address_rank'];
                 $aResults[$aResult['place_id']] = $oResult;
             }
         }
@@ -759,7 +751,7 @@ class SearchDescription
         $sHouseNumberRegex = '\\\\m'.$this->sHouseNumber.'\\\\M';
         $sSQL = 'SELECT place_id FROM placex ';
         $sSQL .= 'WHERE parent_place_id in ('.$sPlaceIDs.')';
-        $sSQL .= "  AND transliteration(housenumber) ~* E'".$sHouseNumberRegex."'";
+        $sSQL .= "  AND housenumber ~* E'".$sHouseNumberRegex."'";
         $sSQL .= $this->oContext->excludeSQL(' AND place_id');
 
         Debug::printSQL($sSQL);
@@ -1027,7 +1019,7 @@ class SearchDescription
                 'Name terms (stop words)' => $this->aNameNonSearch,
                 'Address terms' => $this->aAddress,
                 'Address terms (stop words)' => $this->aAddressNonSearch,
-                'Address terms (full words)' => $this->aFullNameAddress,
+                'Address terms (full words)' => $this->aFullNameAddress ?? '',
                 'Special search' => $this->iOperator,
                 'Class' => $this->sClass,
                 'Type' => $this->sType,
@@ -1039,7 +1031,7 @@ class SearchDescription
     public function dumpAsHtmlTableRow(&$aWordIDs)
     {
         $kf = function ($k) use (&$aWordIDs) {
-            return $aWordIDs[$k];
+            return $aWordIDs[$k] ?? '['.$k.']';
         };
 
         echo '<tr>';

lib-php/Shell.php

@@ -7,7 +7,7 @@ class Shell
     public function __construct($sBaseCmd, ...$aParams)
     {
         if (!$sBaseCmd) {
-            throw new Exception('Command missing in new() call');
+            throw new \Exception('Command missing in new() call');
         }
         $this->baseCmd = $sBaseCmd;
         $this->aParams = array();
@@ -48,7 +48,7 @@ class Shell
         return join(' ', $aEscaped);
     }
 
-    public function run()
+    public function run($bExitOnFail = false)
     {
         $sCmd = $this->escapedCmd();
         // $aEnv does not need escaping, proc_open seems to handle it fine
@@ -67,6 +67,11 @@ class Shell
         fclose($aPipes[0]); // no stdin
 
         $iStat = proc_close($hProc);
+
+        if ($iStat != 0 && $bExitOnFail) {
+            exit($iStat);
+        }
+
         return $iStat;
     }
 

lib-php/Status.php

@@ -56,4 +56,10 @@ class Status
 
         return $iDataDateEpoch;
     }
+
+    public function databaseVersion()
+    {
+        $sSQL = 'SELECT value FROM nominatim_properties WHERE property = \'database_version\'';
+        return $this->oDB->getOne($sSQL);
+    }
 }

lib-php/TokenList.php

@@ -2,12 +2,12 @@
 
 namespace Nominatim;
 
-require_once(CONST_BasePath.'/lib/TokenCountry.php');
-require_once(CONST_BasePath.'/lib/TokenHousenumber.php');
-require_once(CONST_BasePath.'/lib/TokenPostcode.php');
-require_once(CONST_BasePath.'/lib/TokenSpecialTerm.php');
-require_once(CONST_BasePath.'/lib/TokenWord.php');
-require_once(CONST_BasePath.'/lib/SpecialSearchOperator.php');
+require_once(CONST_LibDir.'/TokenCountry.php');
+require_once(CONST_LibDir.'/TokenHousenumber.php');
+require_once(CONST_LibDir.'/TokenPostcode.php');
+require_once(CONST_LibDir.'/TokenSpecialTerm.php');
+require_once(CONST_LibDir.'/TokenWord.php');
+require_once(CONST_LibDir.'/SpecialSearchOperator.php');
 
 /**
  * Saves information about the tokens that appear in a search query.

lib-php/TokenSpecialTerm.php

@@ -2,7 +2,7 @@
 
 namespace Nominatim\Token;
 
-require_once(CONST_BasePath.'/lib/SpecialSearchOperator.php');
+require_once(CONST_LibDir.'/SpecialSearchOperator.php');
 
 /**
  * A word token describing a place type.

lib-php/admin/check_import_finished.php

@@ -0,0 +1,10 @@
+<?php
+@define('CONST_LibDir', dirname(dirname(__FILE__)));
+
+require_once(CONST_LibDir.'/init-cmd.php');
+
+loadSettings(getcwd());
+
+(new \Nominatim\Shell(getSetting('NOMINATIM_TOOL')))
+    ->addParams('admin', '--check-database')
+    ->run();

lib-php/admin/country_languages.php

@@ -1,6 +1,7 @@
 <?php
+@define('CONST_LibDir', dirname(dirname(__FILE__)));
 
-require_once(CONST_BasePath.'/lib/init-cmd.php');
+require_once(CONST_LibDir.'/init-cmd.php');
 
 ini_set('memory_limit', '800M');
 ini_set('display_errors', 'stderr');
@@ -11,10 +12,12 @@ $aCMDOptions
     array('help', 'h', 0, 1, 0, 0, false, 'Show Help'),
     array('quiet', 'q', 0, 1, 0, 0, 'bool', 'Quiet output'),
     array('verbose', 'v', 0, 1, 0, 0, 'bool', 'Verbose output'),
+    array('project-dir', '', 0, 1, 1, 1, 'realpath', 'Base directory of the Nominatim installation (default: .)'),
    );
 getCmdOpt($_SERVER['argv'], $aCMDOptions, $aCMDResult, true, true);
 
-include(CONST_Phrase_Config);
+loadSettings($aCMDResult['project-dir'] ?? getcwd());
+setupHTTPProxy();
 
 if (true) {
     $sURL = 'https://wiki.openstreetmap.org/wiki/Special:Export/Nominatim/Country_Codes';

lib-php/admin/export.php

@@ -1,10 +1,11 @@
 <?php
+    @define('CONST_LibDir', dirname(dirname(__FILE__)));
     // Script to extract structured city and street data
     // from a running nominatim instance as CSV data
 
 
-    require_once(CONST_BasePath.'/lib/init-cmd.php');
-    require_once(CONST_BasePath.'/lib/ParameterParser.php');
+    require_once(CONST_LibDir.'/init-cmd.php');
+    require_once(CONST_LibDir.'/ParameterParser.php');
     ini_set('memory_limit', '800M');
 
     $aCMDOptions = array(
@@ -21,6 +22,7 @@
                     array('restrict-to-osm-node', '', 0, 1, 1, 1, 'int', 'Export only objects that are children of this OSM node'),
                     array('restrict-to-osm-way', '', 0, 1, 1, 1, 'int', 'Export only objects that are children of this OSM way'),
                     array('restrict-to-osm-relation', '', 0, 1, 1, 1, 'int', 'Export only objects that are children of this OSM relation'),
+                    array('project-dir', '', 0, 1, 1, 1, 'realpath', 'Base directory of the Nominatim installation (default: .)'),
                     "\nAddress ranks: continent, country, state, county, city, suburb, street, path",
                     'Additional output types: postcode, placeid (placeid for each object)',
                     "\noutput-format must be a semicolon-separated list of address ranks. Multiple ranks",
@@ -30,6 +32,8 @@
                    );
     getCmdOpt($_SERVER['argv'], $aCMDOptions, $aCMDResult, true, true);
 
+    loadSettings($aCMDResult['project-dir'] ?? getcwd());
+
     $aRankmap = array(
                  'continent' => 1,
                  'country' => 4,

lib-php/admin/query.php

@@ -1,8 +1,9 @@
 <?php
+@define('CONST_LibDir', dirname(dirname(__FILE__)));
 
-require_once(CONST_BasePath.'/lib/init-cmd.php');
-require_once(CONST_BasePath.'/lib/Geocode.php');
-require_once(CONST_BasePath.'/lib/ParameterParser.php');
+require_once(CONST_LibDir.'/init-cmd.php');
+require_once(CONST_LibDir.'/Geocode.php');
+require_once(CONST_LibDir.'/ParameterParser.php');
 ini_set('memory_limit', '800M');
 
 $aCMDOptions
@@ -28,10 +29,30 @@ $aCMDOptions
    array('exclude_place_ids', '', 0, 1, 1, 1, 'string', 'Comma-separated list of place ids to exclude from results'),
    array('featureType', '', 0, 1, 1, 1, 'string', 'Restrict results to certain features (country, state,city,settlement)'),
    array('countrycodes', '', 0, 1, 1, 1, 'string', 'Comma-separated list of countries to restrict search to'),
-   array('viewbox', '', 0, 1, 1, 1, 'string', 'Prefer results in given view box')
+   array('viewbox', '', 0, 1, 1, 1, 'string', 'Prefer results in given view box'),
+
+   array('project-dir', '', 0, 1, 1, 1, 'realpath', 'Base directory of the Nominatim installation (default: .)'),
   );
 getCmdOpt($_SERVER['argv'], $aCMDOptions, $aCMDResult, true, true);
 
+loadSettings($aCMDResult['project-dir'] ?? getcwd());
+
+@define('CONST_Database_DSN', getSetting('DATABASE_DSN'));
+@define('CONST_Default_Language', getSetting('DEFAULT_LANGUAGE', false));
+@define('CONST_Log_DB', getSettingBool('LOG_DB'));
+@define('CONST_Log_File', getSetting('LOG_FILE', false));
+@define('CONST_Max_Word_Frequency', getSetting('MAX_WORD_FREQUENCY'));
+@define('CONST_NoAccessControl', getSettingBool('CORS_NOACCESSCONTROL'));
+@define('CONST_Places_Max_ID_count', getSetting('LOOKUP_MAX_COUNT'));
+@define('CONST_PolygonOutput_MaximumTypes', getSetting('POLYGON_OUTPUT_MAX_TYPES'));
+@define('CONST_Search_BatchMode', getSettingBool('SEARCH_BATCH_MODE'));
+@define('CONST_Search_NameOnlySearchFrequencyThreshold', getSetting('SEARCH_NAME_ONLY_THRESHOLD'));
+@define('CONST_Term_Normalization_Rules', getSetting('TERM_NORMALIZATION'));
+@define('CONST_Use_Aux_Location_data', getSettingBool('USE_AUX_LOCATION_DATA'));
+@define('CONST_Use_US_Tiger_Data', getSettingBool('USE_US_TIGER_DATA'));
+@define('CONST_MapIcon_URL', getSetting('MAPICON_URL', false));
+
+
 $oDB = new Nominatim\DB;
 $oDB->connect();
 

lib-php/admin/setup.php

@@ -1,8 +1,9 @@
 <?php
+@define('CONST_LibDir', dirname(dirname(__FILE__)));
 
-require_once(CONST_BasePath.'/lib/init-cmd.php');
-require_once(CONST_BasePath.'/lib/setup/SetupClass.php');
-require_once(CONST_BasePath.'/lib/setup_functions.php');
+require_once(CONST_LibDir.'/init-cmd.php');
+require_once(CONST_LibDir.'/setup/SetupClass.php');
+require_once(CONST_LibDir.'/setup_functions.php');
 ini_set('memory_limit', '800M');
 
 use Nominatim\Setup\SetupFunctions as SetupFunctions;
@@ -44,13 +45,40 @@ $aCMDOptions
    array('create-country-names', '', 0, 1, 0, 0, 'bool', 'Create default list of searchable country names'),
    array('drop', '', 0, 1, 0, 0, 'bool', 'Drop tables needed for updates, making the database readonly (EXPERIMENTAL)'),
    array('setup-website', '', 0, 1, 0, 0, 'bool', 'Used to compile environment variables for the website'),
+   array('project-dir', '', 0, 1, 1, 1, 'realpath', 'Base directory of the Nominatim installation (default: .)'),
   );
 
 // $aCMDOptions passed to getCmdOpt by reference
 getCmdOpt($_SERVER['argv'], $aCMDOptions, $aCMDResult, true, true);
 
+loadSettings($aCMDResult['project-dir'] ?? getcwd());
+setupHTTPProxy();
+
 $bDidSomething = false;
 
+$oNominatimCmd = new \Nominatim\Shell(getSetting('NOMINATIM_TOOL'));
+
+// by default, use all but one processor, but never more than 15.
+$iInstances = max(1, $aCMDResult['threads'] ?? (min(16, getProcessorCount()) - 1));
+
+function run($oCmd)
+{
+    global $iInstances;
+    global $aCMDResult;
+    $oCmd->addParams('--threads', $iInstances);
+    if ($aCMDResult['ignore-errors'] ?? false) {
+        $oCmd->addParams('--ignore-errors');
+    }
+    if ($aCMDResult['quiet'] ?? false) {
+        $oCmd->addParams('--quiet');
+    }
+    if ($aCMDResult['verbose'] ?? false) {
+        $oCmd->addParams('--verbose');
+    }
+    $oCmd->run(true);
+}
+
+
 //*******************************************************
 // Making some sanity check:
 // Check if osm-file is set and points to a valid file
@@ -67,54 +95,73 @@ $oSetup = new SetupFunctions($aCMDResult);
 // go through complete process if 'all' is selected or start selected functions
 if ($aCMDResult['create-db'] || $aCMDResult['all']) {
     $bDidSomething = true;
-    $oSetup->createDB();
+    run((clone($oNominatimCmd))->addParams('transition', '--create-db'));
 }
 
 if ($aCMDResult['setup-db'] || $aCMDResult['all']) {
     $bDidSomething = true;
-    $oSetup->setupDB();
+    $oCmd = (clone($oNominatimCmd))->addParams('transition', '--setup-db');
+
+    if ($aCMDResult['no-partitions'] ?? false) {
+        $oCmd->addParams('--no-partitions');
+    }
+
+    run($oCmd);
 }
 
 if ($aCMDResult['import-data'] || $aCMDResult['all']) {
     $bDidSomething = true;
-    $oSetup->importData($aCMDResult['osm-file']);
+    $oCmd = (clone($oNominatimCmd))
+        ->addParams('transition', '--import-data')
+        ->addParams('--osm-file', $aCMDResult['osm-file']);
+    if ($aCMDResult['drop'] ?? false) {
+        $oCmd->addParams('--drop');
+    }
+
+    run($oCmd);
 }
 
 if ($aCMDResult['create-functions'] || $aCMDResult['all']) {
     $bDidSomething = true;
-    $oSetup->createFunctions();
+    $oSetup->createSqlFunctions();
 }
 
 if ($aCMDResult['create-tables'] || $aCMDResult['all']) {
     $bDidSomething = true;
-    $oSetup->createTables($aCMDResult['reverse-only']);
-    $oSetup->createFunctions();
-    $oSetup->createTableTriggers();
+    $oCmd = (clone($oNominatimCmd))->addParams('transition', '--create-tables');
+
+    if ($aCMDResult['reverse-only'] ?? false) {
+        $oCmd->addParams('--reverse-only');
+    }
+
+    run($oCmd);
 }
 
 if ($aCMDResult['create-partition-tables'] || $aCMDResult['all']) {
     $bDidSomething = true;
-    $oSetup->createPartitionTables();
+    run((clone($oNominatimCmd))->addParams('transition', '--create-partition-tables'));
 }
 
 if ($aCMDResult['create-partition-functions'] || $aCMDResult['all']) {
     $bDidSomething = true;
-    $oSetup->createPartitionFunctions();
+    $oSetup->createSqlFunctions(); // also create partition functions
 }
 
 if ($aCMDResult['import-wikipedia-articles'] || $aCMDResult['all']) {
     $bDidSomething = true;
-    $oSetup->importWikipediaArticles();
+    // ignore errors!
+    (clone($oNominatimCmd))->addParams('refresh', '--wiki-data')->run();
 }
 
 if ($aCMDResult['load-data'] || $aCMDResult['all']) {
     $bDidSomething = true;
-    $oSetup->loadData($aCMDResult['disable-token-precalc']);
+    run((clone($oNominatimCmd))->addParams('transition', '--load-data'));
 }
 
 if ($aCMDResult['import-tiger-data']) {
     $bDidSomething = true;
-    $oSetup->importTigerData();
+    $sTigerPath = getSetting('TIGER_DATA_PATH', CONST_InstallDir.'/tiger');
+    run((clone($oNominatimCmd))->addParams('transition', '--tiger-data', $sTigerPath));
 }
 
 if ($aCMDResult['calculate-postcodes'] || $aCMDResult['all']) {
@@ -124,27 +171,39 @@ if ($aCMDResult['calculate-postcodes'] || $aCMDResult['all']) {
 
 if ($aCMDResult['index'] || $aCMDResult['all']) {
     $bDidSomething = true;
-    $oSetup->index($aCMDResult['index-noanalyse']);
+    $oCmd = (clone($oNominatimCmd))->addParams('transition', '--index');
+    if ($aCMDResult['index-noanalyse'] ?? false) {
+        $oCmd->addParams('--no-analyse');
+    }
+
+    run($oCmd);
 }
 
 if ($aCMDResult['drop']) {
     $bDidSomething = true;
-    $oSetup->drop($aCMDResult);
+    run((clone($oNominatimCmd))->addParams('freeze'));
 }
 
 if ($aCMDResult['create-search-indices'] || $aCMDResult['all']) {
     $bDidSomething = true;
-    $oSetup->createSearchIndices();
+
+    $oCmd = (clone($oNominatimCmd))->addParams('transition', '--create-search-indices');
+
+    if ($aCMDResult['drop'] ?? false) {
+        $oCmd->addParams('--drop');
+    }
+
+    run($oCmd);
 }
 
 if ($aCMDResult['create-country-names'] || $aCMDResult['all']) {
     $bDidSomething = true;
-    $oSetup->createCountryNames($aCMDResult);
+    run(clone($oNominatimCmd))->addParams('transition', '--create-country-names');
 }
 
 if ($aCMDResult['setup-website'] || $aCMDResult['all']) {
     $bDidSomething = true;
-    $oSetup->setupWebsite();
+    run((clone($oNominatimCmd))->addParams('refresh', '--website'));
 }
 
 // ******************************************************

lib-php/admin/specialphrases.php

@@ -0,0 +1,11 @@
+
+<?php
+@define('CONST_LibDir', dirname(dirname(__FILE__)));
+
+require_once(CONST_LibDir.'/init-cmd.php');
+
+loadSettings(getcwd());
+
+(new \Nominatim\Shell(getSetting('NOMINATIM_TOOL')))
+    ->addParams('special-phrases', '--import-from-wiki')
+    ->run();

lib-php/admin/update.php

@@ -0,0 +1,236 @@
+<?php
+@define('CONST_LibDir', dirname(dirname(__FILE__)));
+
+require_once(CONST_LibDir.'/init-cmd.php');
+require_once(CONST_LibDir.'/setup_functions.php');
+require_once(CONST_LibDir.'/setup/SetupClass.php');
+
+ini_set('memory_limit', '800M');
+
+use Nominatim\Setup\SetupFunctions as SetupFunctions;
+
+// (long-opt, short-opt, min-occurs, max-occurs, num-arguments, num-arguments, type, help)
+$aCMDOptions
+= array(
+   'Import / update / index osm data',
+   array('help', 'h', 0, 1, 0, 0, false, 'Show Help'),
+   array('quiet', 'q', 0, 1, 0, 0, 'bool', 'Quiet output'),
+   array('verbose', 'v', 0, 1, 0, 0, 'bool', 'Verbose output'),
+
+   array('init-updates', '', 0, 1, 0, 0, 'bool', 'Set up database for updating'),
+   array('check-for-updates', '', 0, 1, 0, 0, 'bool', 'Check if new updates are available'),
+   array('no-update-functions', '', 0, 1, 0, 0, 'bool', 'Do not update trigger functions to support differential updates (assuming the diff update logic is already present)'),
+   array('import-osmosis', '', 0, 1, 0, 0, 'bool', 'Import updates once'),
+   array('import-osmosis-all', '', 0, 1, 0, 0, 'bool', 'Import updates forever'),
+   array('no-index', '', 0, 1, 0, 0, 'bool', 'Do not index the new data'),
+
+   array('calculate-postcodes', '', 0, 1, 0, 0, 'bool', 'Update postcode centroid table'),
+
+   array('import-file', '', 0, 1, 1, 1, 'realpath', 'Re-import data from an OSM file'),
+   array('import-diff', '', 0, 1, 1, 1, 'realpath', 'Import a diff (osc) file from local file system'),
+   array('osm2pgsql-cache', '', 0, 1, 1, 1, 'int', 'Cache size used by osm2pgsql'),
+
+   array('import-node', '', 0, 1, 1, 1, 'int', 'Re-import node'),
+   array('import-way', '', 0, 1, 1, 1, 'int', 'Re-import way'),
+   array('import-relation', '', 0, 1, 1, 1, 'int', 'Re-import relation'),
+   array('import-from-main-api', '', 0, 1, 0, 0, 'bool', 'Use OSM API instead of Overpass to download objects'),
+
+   array('index', '', 0, 1, 0, 0, 'bool', 'Index'),
+   array('index-rank', '', 0, 1, 1, 1, 'int', 'Rank to start indexing from'),
+   array('index-instances', '', 0, 1, 1, 1, 'int', 'Number of indexing instances (threads)'),
+
+   array('recompute-word-counts', '', 0, 1, 0, 0, 'bool', 'Compute frequency of full-word search terms'),
+   array('update-address-levels', '', 0, 1, 0, 0, 'bool', 'Reimport address level configuration (EXPERT)'),
+   array('recompute-importance', '', 0, 1, 0, 0, 'bool', 'Recompute place importances'),
+
+   array('project-dir', '', 0, 1, 1, 1, 'realpath', 'Base directory of the Nominatim installation (default: .)'),
+  );
+
+getCmdOpt($_SERVER['argv'], $aCMDOptions, $aResult, true, true);
+
+loadSettings($aCMDResult['project-dir'] ?? getcwd());
+setupHTTPProxy();
+
+if (!isset($aResult['index-instances'])) $aResult['index-instances'] = 1;
+if (!isset($aResult['index-rank'])) $aResult['index-rank'] = 0;
+
+date_default_timezone_set('Etc/UTC');
+
+$oDB = new Nominatim\DB();
+$oDB->connect();
+$fPostgresVersion = $oDB->getPostgresVersion();
+
+$aDSNInfo = Nominatim\DB::parseDSN(getSetting('DATABASE_DSN'));
+if (!isset($aDSNInfo['port']) || !$aDSNInfo['port']) $aDSNInfo['port'] = 5432;
+
+// cache memory to be used by osm2pgsql, should not be more than the available memory
+$iCacheMemory = (isset($aResult['osm2pgsql-cache'])?$aResult['osm2pgsql-cache']:2000);
+if ($iCacheMemory + 500 > getTotalMemoryMB()) {
+    $iCacheMemory = getCacheMemoryMB();
+    echo "WARNING: resetting cache memory to $iCacheMemory\n";
+}
+
+$oOsm2pgsqlCmd = (new \Nominatim\Shell(getOsm2pgsqlBinary()))
+                 ->addParams('--hstore')
+                 ->addParams('--latlong')
+                 ->addParams('--append')
+                 ->addParams('--slim')
+                 ->addParams('--with-forward-dependencies', 'false')
+                 ->addParams('--log-progress', 'true')
+                 ->addParams('--number-processes', 1)
+                 ->addParams('--cache', $iCacheMemory)
+                 ->addParams('--output', 'gazetteer')
+                 ->addParams('--style', getImportStyle())
+                 ->addParams('--database', $aDSNInfo['database'])
+                 ->addParams('--port', $aDSNInfo['port']);
+
+if (isset($aDSNInfo['hostspec']) && $aDSNInfo['hostspec']) {
+    $oOsm2pgsqlCmd->addParams('--host', $aDSNInfo['hostspec']);
+}
+if (isset($aDSNInfo['username']) && $aDSNInfo['username']) {
+    $oOsm2pgsqlCmd->addParams('--user', $aDSNInfo['username']);
+}
+if (isset($aDSNInfo['password']) && $aDSNInfo['password']) {
+    $oOsm2pgsqlCmd->addEnvPair('PGPASSWORD', $aDSNInfo['password']);
+}
+if (getSetting('FLATNODE_FILE')) {
+    $oOsm2pgsqlCmd->addParams('--flat-nodes', getSetting('FLATNODE_FILE'));
+}
+if ($fPostgresVersion >= 11.0) {
+    $oOsm2pgsqlCmd->addEnvPair(
+        'PGOPTIONS',
+        '-c jit=off -c max_parallel_workers_per_gather=0'
+    );
+}
+
+$oNominatimCmd = new \Nominatim\Shell(getSetting('NOMINATIM_TOOL'));
+
+function run($oCmd)
+{
+    global $aCMDResult;
+    if ($aCMDResult['quiet'] ?? false) {
+        $oCmd->addParams('--quiet');
+    }
+    if ($aCMDResult['verbose'] ?? false) {
+        $oCmd->addParams('--verbose');
+    }
+    $oCmd->run(true);
+}
+
+
+if ($aResult['init-updates']) {
+    $oCmd = (clone($oNominatimCmd))->addParams('replication', '--init');
+
+    if ($aResult['no-update-functions']) {
+        $oCmd->addParams('--no-update-functions');
+    }
+
+    run($oCmd);
+}
+
+if ($aResult['check-for-updates']) {
+    exit((clone($oNominatimCmd))->addParams('replication', '--check-for-updates')->run());
+}
+
+if (isset($aResult['import-diff']) || isset($aResult['import-file'])) {
+    // import diffs and files directly (e.g. from osmosis --rri)
+    $sNextFile = isset($aResult['import-diff']) ? $aResult['import-diff'] : $aResult['import-file'];
+
+    if (!file_exists($sNextFile)) {
+        fail("Cannot open $sNextFile\n");
+    }
+
+    // Import the file
+    $oCMD = (clone $oOsm2pgsqlCmd)->addParams($sNextFile);
+    echo $oCMD->escapedCmd()."\n";
+    $iRet = $oCMD->run();
+
+    if ($iRet) {
+        fail("Error from osm2pgsql, $iRet\n");
+    }
+
+    // Don't update the import status - we don't know what this file contains
+}
+
+if ($aResult['calculate-postcodes']) {
+    run((clone($oNominatimCmd))->addParams('refresh', '--postcodes'));
+}
+
+$sTemporaryFile = CONST_InstallDir.'/osmosischange.osc';
+$bHaveDiff = false;
+$bUseOSMApi = isset($aResult['import-from-main-api']) && $aResult['import-from-main-api'];
+$sContentURL = '';
+if (isset($aResult['import-node']) && $aResult['import-node']) {
+    if ($bUseOSMApi) {
+        $sContentURL = 'https://www.openstreetmap.org/api/0.6/node/'.$aResult['import-node'];
+    } else {
+        $sContentURL = 'https://overpass-api.de/api/interpreter?data=node('.$aResult['import-node'].');out%20meta;';
+    }
+}
+
+if (isset($aResult['import-way']) && $aResult['import-way']) {
+    if ($bUseOSMApi) {
+        $sContentURL = 'https://www.openstreetmap.org/api/0.6/way/'.$aResult['import-way'].'/full';
+    } else {
+        $sContentURL = 'https://overpass-api.de/api/interpreter?data=(way('.$aResult['import-way'].');%3E;);out%20meta;';
+    }
+}
+
+if (isset($aResult['import-relation']) && $aResult['import-relation']) {
+    if ($bUseOSMApi) {
+        $sContentURL = 'https://www.openstreetmap.org/api/0.6/relation/'.$aResult['import-relation'].'/full';
+    } else {
+        $sContentURL = 'https://overpass-api.de/api/interpreter?data=(rel(id:'.$aResult['import-relation'].');%3E;);out%20meta;';
+    }
+}
+
+if ($sContentURL) {
+    file_put_contents($sTemporaryFile, file_get_contents($sContentURL));
+    $bHaveDiff = true;
+}
+
+if ($bHaveDiff) {
+    // import generated change file
+
+    $oCMD = (clone $oOsm2pgsqlCmd)->addParams($sTemporaryFile);
+    echo $oCMD->escapedCmd()."\n";
+
+    $iRet = $oCMD->run();
+    if ($iRet) {
+        fail("osm2pgsql exited with error level $iRet\n");
+    }
+}
+
+if ($aResult['recompute-word-counts']) {
+    run((clone($oNominatimCmd))->addParams('refresh', '--word-counts'));
+}
+
+if ($aResult['index']) {
+    run((clone $oNominatimCmd)
+        ->addParams('index', '--minrank', $aResult['index-rank'])
+        ->addParams('--threads', $aResult['index-instances']));
+}
+
+if ($aResult['update-address-levels']) {
+    run((clone($oNominatimCmd))->addParams('refresh', '--address-levels'));
+}
+
+if ($aResult['recompute-importance']) {
+    run((clone($oNominatimCmd))->addParams('refresh', '--importance'));
+}
+
+if ($aResult['import-osmosis'] || $aResult['import-osmosis-all']) {
+    $oCmd = (clone($oNominatimCmd))
+              ->addParams('replication')
+              ->addParams('--threads', $aResult['index-instances']);
+
+    if (!$aResult['import-osmosis-all']) {
+        $oCmd->addParams('--once');
+    }
+
+    if ($aResult['no-index']) {
+        $oCmd->addParams('--no-index');
+    }
+
+    run($oCmd);
+}

lib-php/admin/warm.php

@@ -1,6 +1,12 @@
 <?php
+@define('CONST_LibDir', dirname(dirname(__FILE__)));
+
+require_once(CONST_LibDir.'/init-cmd.php');
+require_once(CONST_LibDir.'/log.php');
+require_once(CONST_LibDir.'/Geocode.php');
+require_once(CONST_LibDir.'/PlaceLookup.php');
+require_once(CONST_LibDir.'/ReverseGeocode.php');
 
-require_once(CONST_BasePath.'/lib/init-cmd.php');
 ini_set('memory_limit', '800M');
 
 $aCMDOptions = array(
@@ -10,13 +16,27 @@ $aCMDOptions = array(
                 array('verbose', 'v', 0, 1, 0, 0, 'bool', 'Verbose output'),
                 array('reverse-only', '', 0, 1, 0, 0, 'bool', 'Warm reverse only'),
                 array('search-only', '', 0, 1, 0, 0, 'bool', 'Warm search only'),
+                array('project-dir', '', 0, 1, 1, 1, 'realpath', 'Base directory of the Nominatim installation (default: .)'),
                );
 getCmdOpt($_SERVER['argv'], $aCMDOptions, $aResult, true, true);
 
-require_once(CONST_BasePath.'/lib/log.php');
-require_once(CONST_BasePath.'/lib/Geocode.php');
-require_once(CONST_BasePath.'/lib/PlaceLookup.php');
-require_once(CONST_BasePath.'/lib/ReverseGeocode.php');
+loadSettings($aCMDResult['project-dir'] ?? getcwd());
+
+@define('CONST_Database_DSN', getSetting('DATABASE_DSN'));
+@define('CONST_Default_Language', getSetting('DEFAULT_LANGUAGE', false));
+@define('CONST_Log_DB', getSettingBool('LOG_DB'));
+@define('CONST_Log_File', getSetting('LOG_FILE', false));
+@define('CONST_Max_Word_Frequency', getSetting('MAX_WORD_FREQUENCY'));
+@define('CONST_NoAccessControl', getSettingBool('CORS_NOACCESSCONTROL'));
+@define('CONST_Places_Max_ID_count', getSetting('LOOKUP_MAX_COUNT'));
+@define('CONST_PolygonOutput_MaximumTypes', getSetting('POLYGON_OUTPUT_MAX_TYPES'));
+@define('CONST_Search_BatchMode', getSettingBool('SEARCH_BATCH_MODE'));
+@define('CONST_Search_NameOnlySearchFrequencyThreshold', getSetting('SEARCH_NAME_ONLY_THRESHOLD'));
+@define('CONST_Term_Normalization_Rules', getSetting('TERM_NORMALIZATION'));
+@define('CONST_Use_Aux_Location_data', getSettingBool('USE_AUX_LOCATION_DATA'));
+@define('CONST_Use_US_Tiger_Data', getSettingBool('USE_US_TIGER_DATA'));
+@define('CONST_MapIcon_URL', getSetting('MAPICON_URL', false));
+
 
 $oDB = new Nominatim\DB();
 $oDB->connect();

lib-php/cmd.php

@@ -1,6 +1,6 @@
 <?php
 
-require_once(CONST_BasePath.'/lib/Shell.php');
+require_once(CONST_LibDir.'/Shell.php');
 
 function getCmdOpt($aArg, $aSpec, &$aResult, $bExitOnError = false, $bExitOnUnknown = false)
 {
@@ -147,7 +147,7 @@ function repeatWarnings()
 function runSQLScript($sScript, $bfatal = true, $bVerbose = false, $bIgnoreErrors = false)
 {
     // Convert database DSN to psql parameters
-    $aDSNInfo = \Nominatim\DB::parseDSN(CONST_Database_DSN);
+    $aDSNInfo = \Nominatim\DB::parseDSN(getSetting('DATABASE_DSN'));
     if (!isset($aDSNInfo['port']) || !$aDSNInfo['port']) $aDSNInfo['port'] = 5432;
 
     $oCmd = new \Nominatim\Shell('psql');
@@ -195,3 +195,30 @@ function runSQLScript($sScript, $bfatal = true, $bVerbose = false, $bIgnoreError
         fail("pgsql returned with error code ($iReturn)");
     }
 }
+
+function setupHTTPProxy()
+{
+    if (!getSettingBool('HTTP_PROXY')) {
+        return;
+    }
+
+    $sProxy = 'tcp://'.getSetting('HTTP_PROXY_HOST').':'.getSetting('HTTP_PROXY_PROT');
+    $aHeaders = array();
+
+    $sLogin = getSetting('HTTP_PROXY_LOGIN');
+    $sPassword = getSetting('HTTP_PROXY_PASSWORD');
+
+    if ($sLogin && $sPassword) {
+        $sAuth = base64_encode($sLogin.':'.$sPassword);
+        $aHeaders = array('Proxy-Authorization: Basic '.$sAuth);
+    }
+
+    $aProxyHeader = array(
+                     'proxy' => $sProxy,
+                     'request_fulluri' => true,
+                     'header' => $aHeaders
+                    );
+
+    $aContext = array('http' => $aProxyHeader, 'https' => $aProxyHeader);
+    stream_context_set_default($aContext);
+}

lib-php/dotenv_loader.php

@@ -0,0 +1,13 @@
+<?php
+
+require('Symfony/Component/Dotenv/autoload.php');
+
+function loadDotEnv()
+{
+    $dotenv = new \Symfony\Component\Dotenv\Dotenv();
+    $dotenv->load(CONST_ConfigDir.'/env.defaults');
+
+    if (file_exists('.env')) {
+        $dotenv->load('.env');
+    }
+}

lib-php/init-cmd.php

@@ -0,0 +1,5 @@
+<?php
+
+require_once('init.php');
+require_once('cmd.php');
+require_once('DebugNone.php');

lib-php/init-website.php

@@ -20,7 +20,7 @@ function exception_handler_json($exception)
 {
     http_response_code($exception->getCode());
     header('Content-type: application/json; charset=utf-8');
-    include(CONST_BasePath.'/lib/template/error-json.php');
+    include(CONST_LibDir.'/template/error-json.php');
     exit();
 }
 
@@ -29,7 +29,7 @@ function exception_handler_xml($exception)
     http_response_code($exception->getCode());
     header('Content-type: text/xml; charset=utf-8');
     echo '<?xml version="1.0" encoding="UTF-8" ?>'."\n";
-    include(CONST_BasePath.'/lib/template/error-xml.php');
+    include(CONST_LibDir.'/template/error-xml.php');
     exit();
 }
 

lib-php/init.php

@@ -0,0 +1,4 @@
+<?php
+
+require_once(CONST_LibDir.'/lib.php');
+require_once(CONST_LibDir.'/DB.php');

lib-php/lib.php

@@ -1,5 +1,48 @@
 <?php
 
+function loadSettings($sProjectDir)
+{
+    @define('CONST_InstallDir', $sProjectDir);
+    // Temporary hack to set the direcory via environment instead of
+    // the installed scripts. Neither setting is part of the official
+    // set of settings.
+    defined('CONST_DataDir') or define('CONST_DataDir', $_SERVER['NOMINATIM_DATADIR']);
+    defined('CONST_SqlDir') or define('CONST_SqlDir', $_SERVER['NOMINATIM_SQLDIR']);
+    defined('CONST_ConfigDir') or define('CONST_ConfigDir', $_SERVER['NOMINATIM_CONFIGDIR']);
+    defined('CONST_Default_ModulePath') or define('CONST_Default_ModulePath', $_SERVER['NOMINATIM_DATABASE_MODULE_SRC_PATH']);
+}
+
+function getSetting($sConfName, $sDefault = null)
+{
+    $sValue = $_SERVER['NOMINATIM_'.$sConfName];
+
+    if ($sDefault !== null && !$sValue) {
+        return $sDefault;
+    }
+
+    return $sValue;
+}
+
+function getSettingBool($sConfName)
+{
+    $sVal = strtolower(getSetting($sConfName));
+
+    return strcmp($sVal, 'yes') == 0
+           || strcmp($sVal, 'true') == 0
+           || strcmp($sVal, '1') == 0;
+}
+
+function getSettingConfig($sConfName, $sSystemConfig)
+{
+    $sValue = $_SERVER['NOMINATIM_'.$sConfName];
+
+    if (!$sValue) {
+        return CONST_ConfigDir.'/'.$sSystemConfig;
+    }
+
+    return $sValue;
+}
+
 function fail($sError, $sUserError = false)
 {
     if (!$sUserError) $sUserError = $sError;
@@ -165,17 +208,6 @@ function parseLatLon($sQuery)
     return array($sFound, $fQueryLat, $fQueryLon);
 }
 
-function createPointsAroundCenter($fLon, $fLat, $fRadius)
-{
-    $iSteps = max(8, min(100, ($fRadius * 40000)^2));
-    $fStepSize = (2*pi())/$iSteps;
-    $aPolyPoints = array();
-    for ($f = 0; $f < 2*pi(); $f += $fStepSize) {
-        $aPolyPoints[] = array('', $fLon+($fRadius*sin($f)), $fLat+($fRadius*cos($f)) );
-    }
-    return $aPolyPoints;
-}
-
 function closestHouseNumber($aRow)
 {
     $fHouse = $aRow['startnumber']
@@ -195,25 +227,3 @@ function closestHouseNumber($aRow)
 
     return max(min($aRow['endnumber'], $iHn), $aRow['startnumber']);
 }
-
-function getSearchRankLabel($iRank)
-{
-    if (!isset($iRank)) return 'unknown';
-    if ($iRank < 2) return 'continent';
-    if ($iRank < 4) return 'sea';
-    if ($iRank < 8) return 'country';
-    if ($iRank < 12) return 'state';
-    if ($iRank < 16) return 'county';
-    if ($iRank == 16) return 'city';
-    if ($iRank == 17) return 'town / island';
-    if ($iRank == 18) return 'village / hamlet';
-    if ($iRank == 20) return 'suburb';
-    if ($iRank == 21) return 'postcode area';
-    if ($iRank == 22) return 'croft / farm / locality / islet';
-    if ($iRank == 23) return 'postcode area';
-    if ($iRank == 25) return 'postcode point';
-    if ($iRank == 26) return 'street / major landmark';
-    if ($iRank == 27) return 'minory street / path';
-    if ($iRank == 28) return 'house / building';
-    return 'other: ' . $iRank;
-}

lib-php/migration/PhraseSettingsToJson.php

@@ -0,0 +1,19 @@
+<?php
+
+$phpPhraseSettingsFile = $argv[1];
+$jsonPhraseSettingsFile = dirname($phpPhraseSettingsFile).'/'.basename($phpPhraseSettingsFile, '.php').'.json';
+
+if (file_exists($phpPhraseSettingsFile) && !file_exists($jsonPhraseSettingsFile)) {
+    include $phpPhraseSettingsFile;
+
+    $data = array();
+
+    if (isset($aTagsBlacklist))
+        $data['blackList'] = $aTagsBlacklist;
+    if (isset($aTagsWhitelist))
+        $data['whiteList'] = $aTagsWhitelist;
+
+    $jsonFile = fopen($jsonPhraseSettingsFile, 'w');
+    fwrite($jsonFile, json_encode($data));
+    fclose($jsonFile);
+}

lib-php/output.php

@@ -0,0 +1,18 @@
+<?php
+
+
+function formatOSMType($sType, $bIncludeExternal = true)
+{
+    if ($sType == 'N') return 'node';
+    if ($sType == 'W') return 'way';
+    if ($sType == 'R') return 'relation';
+
+    if (!$bIncludeExternal) return '';
+
+    if ($sType == 'T') return 'way';
+    if ($sType == 'I') return 'way';
+
+    // not handled: P, L
+
+    return '';
+}

lib-php/setup/SetupClass.php

@@ -0,0 +1,261 @@
+<?php
+
+namespace Nominatim\Setup;
+
+require_once(CONST_LibDir.'/Shell.php');
+
+class SetupFunctions
+{
+    protected $iInstances;
+    protected $aDSNInfo;
+    protected $bQuiet;
+    protected $bVerbose;
+    protected $sIgnoreErrors;
+    protected $bEnableDiffUpdates;
+    protected $bEnableDebugStatements;
+    protected $bDrop;
+    protected $oDB = null;
+    protected $oNominatimCmd;
+
+    public function __construct(array $aCMDResult)
+    {
+        // by default, use all but one processor, but never more than 15.
+        $this->iInstances = isset($aCMDResult['threads'])
+            ? $aCMDResult['threads']
+            : (min(16, getProcessorCount()) - 1);
+
+        if ($this->iInstances < 1) {
+            $this->iInstances = 1;
+            warn('resetting threads to '.$this->iInstances);
+        }
+
+        // parse database string
+        $this->aDSNInfo = \Nominatim\DB::parseDSN(getSetting('DATABASE_DSN'));
+        if (!isset($this->aDSNInfo['port'])) {
+            $this->aDSNInfo['port'] = 5432;
+        }
+
+        // setting member variables based on command line options stored in $aCMDResult
+        $this->bQuiet = isset($aCMDResult['quiet']) && $aCMDResult['quiet'];
+        $this->bVerbose = $aCMDResult['verbose'];
+
+        //setting default values which are not set by the update.php array
+        if (isset($aCMDResult['ignore-errors'])) {
+            $this->sIgnoreErrors = $aCMDResult['ignore-errors'];
+        } else {
+            $this->sIgnoreErrors = false;
+        }
+        if (isset($aCMDResult['enable-debug-statements'])) {
+            $this->bEnableDebugStatements = $aCMDResult['enable-debug-statements'];
+        } else {
+            $this->bEnableDebugStatements = false;
+        }
+        if (isset($aCMDResult['enable-diff-updates'])) {
+            $this->bEnableDiffUpdates = $aCMDResult['enable-diff-updates'];
+        } else {
+            $this->bEnableDiffUpdates = false;
+        }
+
+        $this->bDrop = isset($aCMDResult['drop']) && $aCMDResult['drop'];
+
+        $this->oNominatimCmd = new \Nominatim\Shell(getSetting('NOMINATIM_TOOL'));
+        if ($this->bQuiet) {
+            $this->oNominatimCmd->addParams('--quiet');
+        }
+        if ($this->bVerbose) {
+            $this->oNominatimCmd->addParams('--verbose');
+        }
+    }
+
+    public function calculatePostcodes($bCMDResultAll)
+    {
+        info('Calculate Postcodes');
+        $this->pgsqlRunScriptFile(CONST_SqlDir.'/postcode_tables.sql');
+
+        $sPostcodeFilename = CONST_InstallDir.'/gb_postcode_data.sql.gz';
+        if (file_exists($sPostcodeFilename)) {
+            $this->pgsqlRunScriptFile($sPostcodeFilename);
+        } else {
+            warn('optional external GB postcode table file ('.$sPostcodeFilename.') not found. Skipping.');
+        }
+
+        $sPostcodeFilename = CONST_InstallDir.'/us_postcode_data.sql.gz';
+        if (file_exists($sPostcodeFilename)) {
+            $this->pgsqlRunScriptFile($sPostcodeFilename);
+        } else {
+            warn('optional external US postcode table file ('.$sPostcodeFilename.') not found. Skipping.');
+        }
+
+
+        $this->db()->exec('TRUNCATE location_postcode');
+
+        $sSQL  = 'INSERT INTO location_postcode';
+        $sSQL .= ' (place_id, indexed_status, country_code, postcode, geometry) ';
+        $sSQL .= "SELECT nextval('seq_place'), 1, country_code,";
+        $sSQL .= "       upper(trim (both ' ' from address->'postcode')) as pc,";
+        $sSQL .= '       ST_Centroid(ST_Collect(ST_Centroid(geometry)))';
+        $sSQL .= '  FROM placex';
+        $sSQL .= " WHERE address ? 'postcode' AND address->'postcode' NOT SIMILAR TO '%(,|;)%'";
+        $sSQL .= '       AND geometry IS NOT null';
+        $sSQL .= ' GROUP BY country_code, pc';
+        $this->db()->exec($sSQL);
+
+        // only add postcodes that are not yet available in OSM
+        $sSQL  = 'INSERT INTO location_postcode';
+        $sSQL .= ' (place_id, indexed_status, country_code, postcode, geometry) ';
+        $sSQL .= "SELECT nextval('seq_place'), 1, 'us', postcode,";
+        $sSQL .= '       ST_SetSRID(ST_Point(x,y),4326)';
+        $sSQL .= '  FROM us_postcode WHERE postcode NOT IN';
+        $sSQL .= '        (SELECT postcode FROM location_postcode';
+        $sSQL .= "          WHERE country_code = 'us')";
+        $this->db()->exec($sSQL);
+
+        // add missing postcodes for GB (if available)
+        $sSQL  = 'INSERT INTO location_postcode';
+        $sSQL .= ' (place_id, indexed_status, country_code, postcode, geometry) ';
+        $sSQL .= "SELECT nextval('seq_place'), 1, 'gb', postcode, geometry";
+        $sSQL .= '  FROM gb_postcode WHERE postcode NOT IN';
+        $sSQL .= '           (SELECT postcode FROM location_postcode';
+        $sSQL .= "             WHERE country_code = 'gb')";
+        $this->db()->exec($sSQL);
+
+        if (!$bCMDResultAll) {
+            $sSQL = "DELETE FROM word WHERE class='place' and type='postcode'";
+            $sSQL .= 'and word NOT IN (SELECT postcode FROM location_postcode)';
+            $this->db()->exec($sSQL);
+        }
+
+        $sSQL = 'SELECT count(getorcreate_postcode_id(v)) FROM ';
+        $sSQL .= '(SELECT distinct(postcode) as v FROM location_postcode) p';
+        $this->db()->exec($sSQL);
+    }
+
+    /**
+     * Return the connection to the database.
+     *
+     * @return Database object.
+     *
+     * Creates a new connection if none exists yet. Otherwise reuses the
+     * already established connection.
+     */
+    private function db()
+    {
+        if (is_null($this->oDB)) {
+            $this->oDB = new \Nominatim\DB();
+            $this->oDB->connect();
+        }
+
+        return $this->oDB;
+    }
+
+    private function pgsqlRunScript($sScript, $bfatal = true)
+    {
+        runSQLScript(
+            $sScript,
+            $bfatal,
+            $this->bVerbose,
+            $this->sIgnoreErrors
+        );
+    }
+
+    public function createSqlFunctions()
+    {
+        $oCmd = (clone($this->oNominatimCmd))
+                ->addParams('refresh', '--functions');
+
+        if (!$this->bEnableDiffUpdates) {
+            $oCmd->addParams('--no-diff-updates');
+        }
+
+        if ($this->bEnableDebugStatements) {
+            $oCmd->addParams('--enable-debug-statements');
+        }
+
+        $oCmd->run(!$this->sIgnoreErrors);
+    }
+
+    private function pgsqlRunScriptFile($sFilename)
+    {
+        if (!file_exists($sFilename)) fail('unable to find '.$sFilename);
+
+        $oCmd = (new \Nominatim\Shell('psql'))
+                ->addParams('--port', $this->aDSNInfo['port'])
+                ->addParams('--dbname', $this->aDSNInfo['database']);
+
+        if (!$this->bVerbose) {
+            $oCmd->addParams('--quiet');
+        }
+        if (isset($this->aDSNInfo['hostspec'])) {
+            $oCmd->addParams('--host', $this->aDSNInfo['hostspec']);
+        }
+        if (isset($this->aDSNInfo['username'])) {
+            $oCmd->addParams('--username', $this->aDSNInfo['username']);
+        }
+        if (isset($this->aDSNInfo['password'])) {
+            $oCmd->addEnvPair('PGPASSWORD', $this->aDSNInfo['password']);
+        }
+        $ahGzipPipes = null;
+        if (preg_match('/\\.gz$/', $sFilename)) {
+            $aDescriptors = array(
+                             0 => array('pipe', 'r'),
+                             1 => array('pipe', 'w'),
+                             2 => array('file', '/dev/null', 'a')
+                            );
+            $oZcatCmd = new \Nominatim\Shell('zcat', $sFilename);
+
+            $hGzipProcess = proc_open($oZcatCmd->escapedCmd(), $aDescriptors, $ahGzipPipes);
+            if (!is_resource($hGzipProcess)) fail('unable to start zcat');
+            $aReadPipe = $ahGzipPipes[1];
+            fclose($ahGzipPipes[0]);
+        } else {
+            $oCmd->addParams('--file', $sFilename);
+            $aReadPipe = array('pipe', 'r');
+        }
+        $aDescriptors = array(
+                         0 => $aReadPipe,
+                         1 => array('pipe', 'w'),
+                         2 => array('file', '/dev/null', 'a')
+                        );
+        $ahPipes = null;
+
+        $hProcess = proc_open($oCmd->escapedCmd(), $aDescriptors, $ahPipes, null, $oCmd->aEnv);
+        if (!is_resource($hProcess)) fail('unable to start pgsql');
+        // TODO: error checking
+        while (!feof($ahPipes[1])) {
+            echo fread($ahPipes[1], 4096);
+        }
+        fclose($ahPipes[1]);
+        $iReturn = proc_close($hProcess);
+        if ($iReturn > 0) {
+            fail("pgsql returned with error code ($iReturn)");
+        }
+        if ($ahGzipPipes) {
+            fclose($ahGzipPipes[1]);
+            proc_close($hGzipProcess);
+        }
+    }
+
+    private function replaceSqlPatterns($sSql)
+    {
+        $sSql = str_replace('{www-user}', getSetting('DATABASE_WEBUSER'), $sSql);
+
+        $aPatterns = array(
+                      '{ts:address-data}' => getSetting('TABLESPACE_ADDRESS_DATA'),
+                      '{ts:address-index}' => getSetting('TABLESPACE_ADDRESS_INDEX'),
+                      '{ts:search-data}' => getSetting('TABLESPACE_SEARCH_DATA'),
+                      '{ts:search-index}' =>  getSetting('TABLESPACE_SEARCH_INDEX'),
+                      '{ts:aux-data}' =>  getSetting('TABLESPACE_AUX_DATA'),
+                      '{ts:aux-index}' =>  getSetting('TABLESPACE_AUX_INDEX')
+        );
+
+        foreach ($aPatterns as $sPattern => $sTablespace) {
+            if ($sTablespace) {
+                $sSql = str_replace($sPattern, 'TABLESPACE "'.$sTablespace.'"', $sSql);
+            } else {
+                $sSql = str_replace($sPattern, '', $sSql);
+            }
+        }
+
+        return $sSql;
+    }
+}

lib-php/setup_functions.php

@@ -0,0 +1,34 @@
+<?php
+
+function checkInFile($sOSMFile)
+{
+    if (!isset($sOSMFile)) {
+        fail('missing --osm-file for data import');
+    }
+
+    if (!file_exists($sOSMFile)) {
+        fail('the path supplied to --osm-file does not exist');
+    }
+
+    if (!is_readable($sOSMFile)) {
+        fail('osm-file "' . $aCMDResult['osm-file'] . '" not readable');
+    }
+}
+
+function getOsm2pgsqlBinary()
+{
+    $sBinary = getSetting('OSM2PGSQL_BINARY');
+
+    return $sBinary ? $sBinary : CONST_Default_Osm2pgsql;
+}
+
+function getImportStyle()
+{
+    $sStyle = getSetting('IMPORT_STYLE');
+
+    if (in_array($sStyle, array('admin', 'street', 'address', 'full', 'extratags'))) {
+        return CONST_ConfigDir.'/import-'.$sStyle.'.style';
+    }
+
+    return $sStyle;
+}

lib-php/website/deletable.php

@@ -1,15 +1,15 @@
 <?php
 
-require_once(CONST_BasePath.'/lib/init-website.php');
-require_once(CONST_BasePath.'/lib/log.php');
-require_once(CONST_BasePath.'/lib/output.php');
+require_once(CONST_LibDir.'/init-website.php');
+require_once(CONST_LibDir.'/log.php');
+require_once(CONST_LibDir.'/output.php');
 ini_set('memory_limit', '200M');
 
 $oParams = new Nominatim\ParameterParser();
 $sOutputFormat = $oParams->getSet('format', array('json'), 'json');
 set_exception_handler_by_format($sOutputFormat);
 
-$oDB = new Nominatim\DB();
+$oDB = new Nominatim\DB(CONST_Database_DSN);
 $oDB->connect();
 
 $sSQL = 'select placex.place_id, country_code,';

lib-php/website/details.php

@@ -1,9 +1,9 @@
 <?php
 
-require_once(CONST_BasePath.'/lib/init-website.php');
-require_once(CONST_BasePath.'/lib/log.php');
-require_once(CONST_BasePath.'/lib/output.php');
-require_once(CONST_BasePath.'/lib/AddressDetails.php');
+require_once(CONST_LibDir.'/init-website.php');
+require_once(CONST_LibDir.'/log.php');
+require_once(CONST_LibDir.'/output.php');
+require_once(CONST_LibDir.'/AddressDetails.php');
 ini_set('memory_limit', '200M');
 
 $oParams = new Nominatim\ParameterParser();
@@ -25,7 +25,7 @@ $bIncludeHierarchy = $oParams->getBool('hierarchy', false);
 $bGroupHierarchy = $oParams->getBool('group_hierarchy', false);
 $bIncludePolygonAsGeoJSON = $oParams->getBool('polygon_geojson', false);
 
-$oDB = new Nominatim\DB();
+$oDB = new Nominatim\DB(CONST_Database_DSN);
 $oDB->connect();
 
 $sLanguagePrefArraySQL = $oDB->getArraySQL($oDB->getDBQuotedList($aLangPrefOrder));
@@ -53,7 +53,7 @@ if ($sOsmType && $iOsmId > 0) {
 
     // Be nice about our error messages for broken geometry
 
-    if (!$sPlaceId) {
+    if (!$sPlaceId && $oDB->tableExists('import_polygon_error')) {
         $sSQL = 'SELECT ';
         $sSQL .= '    osm_type, ';
         $sSQL .= '    osm_id, ';
@@ -77,15 +77,20 @@ if ($sOsmType && $iOsmId > 0) {
                 $aPointDetails['error_x'] = 0;
                 $aPointDetails['error_y'] = 0;
             }
-            include(CONST_BasePath.'/lib/template/details-error-'.$sOutputFormat.'.php');
+            include(CONST_LibDir.'/template/details-error-'.$sOutputFormat.'.php');
             exit;
         }
     }
+
+    if ($sPlaceId === false) {
+        throw new Exception('No place with that OSM ID found.', 404);
+    }
+} else {
+    if ($sPlaceId === false) {
+        userError('Required parameters missing. Need either osmtype/osmid or place_id.');
+    }
 }
 
-
-if ($sPlaceId === false) userError('Please select a place id');
-
 $iPlaceID = (int)$sPlaceId;
 
 if (CONST_Use_US_Tiger_Data) {
@@ -140,11 +145,10 @@ $sSQL .= " WHERE place_id = $iPlaceID";
 $aPointDetails = $oDB->getRow($sSQL, null, 'Could not get details of place object.');
 
 if (!$aPointDetails) {
-    userError('Unknown place id.');
+    throw new Exception('No place with that place ID found.', 404);
 }
 
 $aPointDetails['localname'] = $aPointDetails['localname']?$aPointDetails['localname']:$aPointDetails['housenumber'];
-$aPointDetails['rank_search_label'] = getSearchRankLabel($aPointDetails['rank_search']); // only used in HTML format
 
 // Get all alternative names (languages, etc)
 $sSQL = 'SELECT (each(name)).key,(each(name)).value FROM placex ';
@@ -247,4 +251,4 @@ if ($bIncludeKeywords) {
 
 logEnd($oDB, $hLog, 1);
 
-include(CONST_BasePath.'/lib/template/details-'.$sOutputFormat.'.php');
+include(CONST_LibDir.'/template/details-'.$sOutputFormat.'.php');

lib-php/website/lookup.php

@@ -1,9 +1,9 @@
 <?php
 
-require_once(CONST_BasePath.'/lib/init-website.php');
-require_once(CONST_BasePath.'/lib/log.php');
-require_once(CONST_BasePath.'/lib/PlaceLookup.php');
-require_once(CONST_BasePath.'/lib/output.php');
+require_once(CONST_LibDir.'/init-website.php');
+require_once(CONST_LibDir.'/log.php');
+require_once(CONST_LibDir.'/PlaceLookup.php');
+require_once(CONST_LibDir.'/output.php');
 ini_set('memory_limit', '200M');
 
 $oParams = new Nominatim\ParameterParser();
@@ -15,7 +15,7 @@ set_exception_handler_by_format($sOutputFormat);
 // Preferred language
 $aLangPrefOrder = $oParams->getPreferredLanguages();
 
-$oDB = new Nominatim\DB();
+$oDB = new Nominatim\DB(CONST_Database_DSN);
 $oDB->connect();
 
 $hLog = logStart($oDB, 'place', $_SERVER['QUERY_STRING'], $aLangPrefOrder);
@@ -84,4 +84,4 @@ $sMoreURL = '';
 logEnd($oDB, $hLog, 1);
 
 $sOutputTemplate = ($sOutputFormat == 'jsonv2') ? 'json' : $sOutputFormat;
-include(CONST_BasePath.'/lib/template/search-'.$sOutputTemplate.'.php');
+include(CONST_LibDir.'/template/search-'.$sOutputTemplate.'.php');

lib-php/website/polygons.php

@@ -1,8 +1,8 @@
 <?php
 
-require_once(CONST_BasePath.'/lib/init-website.php');
-require_once(CONST_BasePath.'/lib/log.php');
-require_once(CONST_BasePath.'/lib/output.php');
+require_once(CONST_LibDir.'/init-website.php');
+require_once(CONST_LibDir.'/log.php');
+require_once(CONST_LibDir.'/output.php');
 ini_set('memory_limit', '200M');
 
 $oParams = new Nominatim\ParameterParser();
@@ -13,7 +13,7 @@ $iDays = $oParams->getInt('days', false);
 $bReduced = $oParams->getBool('reduced', false);
 $sClass = $oParams->getString('class', false);
 
-$oDB = new Nominatim\DB();
+$oDB = new Nominatim\DB(CONST_Database_DSN);
 $oDB->connect();
 
 $iTotalBroken = (int) $oDB->getOne('SELECT count(*) FROM import_polygon_error');

lib-php/website/reverse.php

@@ -1,10 +1,10 @@
 <?php
 
-require_once(CONST_BasePath.'/lib/init-website.php');
-require_once(CONST_BasePath.'/lib/log.php');
-require_once(CONST_BasePath.'/lib/PlaceLookup.php');
-require_once(CONST_BasePath.'/lib/ReverseGeocode.php');
-require_once(CONST_BasePath.'/lib/output.php');
+require_once(CONST_LibDir.'/init-website.php');
+require_once(CONST_LibDir.'/log.php');
+require_once(CONST_LibDir.'/PlaceLookup.php');
+require_once(CONST_LibDir.'/ReverseGeocode.php');
+require_once(CONST_LibDir.'/output.php');
 ini_set('memory_limit', '200M');
 
 $oParams = new Nominatim\ParameterParser();
@@ -16,7 +16,7 @@ set_exception_handler_by_format($sOutputFormat);
 // Preferred language
 $aLangPrefOrder = $oParams->getPreferredLanguages();
 
-$oDB = new Nominatim\DB();
+$oDB = new Nominatim\DB(CONST_Database_DSN);
 $oDB->connect();
 
 $hLog = logStart($oDB, 'reverse', $_SERVER['QUERY_STRING'], $aLangPrefOrder);
@@ -84,4 +84,4 @@ if ($sOutputFormat == 'geocodejson') {
 }
 
 $sOutputTemplate = ($sOutputFormat == 'jsonv2') ? 'json' : $sOutputFormat;
-include(CONST_BasePath.'/lib/template/address-'.$sOutputTemplate.'.php');
+include(CONST_LibDir.'/template/address-'.$sOutputTemplate.'.php');

lib-php/website/search.php

@@ -1,12 +1,12 @@
 <?php
 
-require_once(CONST_BasePath.'/lib/init-website.php');
-require_once(CONST_BasePath.'/lib/log.php');
-require_once(CONST_BasePath.'/lib/Geocode.php');
-require_once(CONST_BasePath.'/lib/output.php');
+require_once(CONST_LibDir.'/init-website.php');
+require_once(CONST_LibDir.'/log.php');
+require_once(CONST_LibDir.'/Geocode.php');
+require_once(CONST_LibDir.'/output.php');
 ini_set('memory_limit', '200M');
 
-$oDB = new Nominatim\DB();
+$oDB = new Nominatim\DB(CONST_Database_DSN);
 $oDB->connect();
 $oParams = new Nominatim\ParameterParser();
 
@@ -15,15 +15,6 @@ $oGeocode = new Nominatim\Geocode($oDB);
 $aLangPrefOrder = $oParams->getPreferredLanguages();
 $oGeocode->setLanguagePreference($aLangPrefOrder);
 
-if (CONST_Search_ReversePlanForAll
-    || isset($aLangPrefOrder['name:de'])
-    || isset($aLangPrefOrder['name:ru'])
-    || isset($aLangPrefOrder['name:ja'])
-    || isset($aLangPrefOrder['name:pl'])
-) {
-    $oGeocode->setReverseInPlan(true);
-}
-
 // Format for output
 $sOutputFormat = $oParams->getSet('format', array('xml', 'json', 'jsonv2', 'geojson', 'geocodejson'), 'jsonv2');
 set_exception_handler_by_format($sOutputFormat);
@@ -41,7 +32,7 @@ if (CONST_Search_BatchMode && isset($_GET['batch'])) {
         $aSearchResults = $oBatchGeocode->lookup();
         $aBatchResults[] = $aSearchResults;
     }
-    include(CONST_BasePath.'/lib/template/search-batch-json.php');
+    include(CONST_LibDir.'/template/search-batch-json.php');
     exit;
 }
 
@@ -76,11 +67,11 @@ if (isset($_SERVER['HTTP_ACCEPT_LANGUAGE'])) {
 }
 
 if (isset($_SERVER['REQUEST_SCHEME'])
-    && isset($_SERVER['SERVER_NAME'])
+    && isset($_SERVER['HTTP_HOST'])
     && isset($_SERVER['DOCUMENT_URI'])
 ) {
     $sMoreURL = $_SERVER['REQUEST_SCHEME'].'://'
-                .$_SERVER['SERVER_NAME'].$_SERVER['DOCUMENT_URI'].'/?'
+                .$_SERVER['HTTP_HOST'].$_SERVER['DOCUMENT_URI'].'/?'
                 .http_build_query($aMoreParams);
 } else {
     $sMoreURL = '/search.php'.http_build_query($aMoreParams);
@@ -89,4 +80,4 @@ if (isset($_SERVER['REQUEST_SCHEME'])
 if (CONST_Debug) exit;
 
 $sOutputTemplate = ($sOutputFormat == 'jsonv2') ? 'json' : $sOutputFormat;
-include(CONST_BasePath.'/lib/template/search-'.$sOutputTemplate.'.php');
+include(CONST_LibDir.'/template/search-'.$sOutputTemplate.'.php');

lib-php/website/status.php

@@ -1,13 +1,13 @@
 <?php
 
-require_once(CONST_BasePath.'/lib/init-website.php');
-require_once(CONST_BasePath.'/lib/ParameterParser.php');
-require_once(CONST_BasePath.'/lib/Status.php');
+require_once(CONST_LibDir.'/init-website.php');
+require_once(CONST_LibDir.'/ParameterParser.php');
+require_once(CONST_LibDir.'/Status.php');
 
 $oParams = new Nominatim\ParameterParser();
 $sOutputFormat = $oParams->getSet('format', array('text', 'json'), 'text');
 
-$oDB = new Nominatim\DB();
+$oDB = new Nominatim\DB(CONST_Database_DSN);
 
 if ($sOutputFormat == 'json') {
     header('content-type: application/json; charset=UTF-8');
@@ -37,8 +37,13 @@ if ($sOutputFormat == 'json') {
     $aResponse = array(
                   'status' => 0,
                   'message' => 'OK',
-                  'data_updated' => (new DateTime('@'.$epoch))->format(DateTime::RFC3339)
+                  'data_updated' => (new DateTime('@'.$epoch))->format(DateTime::RFC3339),
+                  'software_version' => CONST_NominatimVersion
                  );
+    $sDatabaseVersion = $oStatus->databaseVersion();
+    if ($sDatabaseVersion) {
+        $aResponse['database_version'] = $sDatabaseVersion;
+    }
     javascript_renderData($aResponse);
 } else {
     echo 'OK';

lib-sql/functions.sql

@@ -0,0 +1,20 @@
+{% include('functions/utils.sql') %}
+{% include('functions/normalization.sql') %}
+{% include('functions/ranking.sql') %}
+{% include('functions/importance.sql') %}
+{% include('functions/address_lookup.sql') %}
+{% include('functions/interpolation.sql') %}
+
+{% if 'place' in db.tables %}
+    {% include 'functions/place_triggers.sql' %}
+{% endif %}
+
+{% if 'placex' in db.tables %}
+    {% include 'functions/placex_triggers.sql' %}
+{% endif %}
+
+{% if 'location_postcode' in db.tables %}
+    {% include 'functions/postcode_triggers.sql' %}
+{% endif %}
+
+{% include('functions/partition-functions.sql') %}

lib-sql/functions/address_lookup.sql

@@ -121,7 +121,7 @@ BEGIN
   END IF;
 
   --then query tiger data
-  -- %NOTIGERDATA% IF 0 THEN
+  {% if config.get_bool('USE_US_TIGER_DATA') %}
   IF place IS NULL AND in_housenumber >= 0 THEN
     SELECT parent_place_id as place_id, 'us' as country_code,
            in_housenumber as housenumber, postcode,
@@ -133,9 +133,10 @@ BEGIN
       WHERE place_id = in_place_id
             AND in_housenumber between startnumber and endnumber;
   END IF;
-  -- %NOTIGERDATA% END IF;
+  {% endif %}
 
-  -- %NOAUXDATA% IF 0 THEN
+  -- then additional data
+  {% if config.get_bool('USE_AUX_LOCATION_DATA') %}
   IF place IS NULL THEN
     SELECT parent_place_id as place_id, 'us' as country_code,
            housenumber, postcode,
@@ -146,7 +147,7 @@ BEGIN
       FROM location_property_aux
       WHERE place_id = in_place_id;
   END IF;
-  -- %NOAUXDATA% END IF;
+  {% endif %}
 
   -- postcode table
   IF place IS NULL THEN
@@ -163,7 +164,10 @@ BEGIN
   -- POI objects in the placex table
   IF place IS NULL THEN
     SELECT parent_place_id as place_id, country_code,
-           housenumber, postcode,
+           coalesce(address->'housenumber',
+                    address->'streetnumber',
+                    address->'conscriptionnumber')::text as housenumber,
+           postcode,
            class, type,
            name, address,
            centroid
@@ -177,7 +181,7 @@ BEGIN
   -- place we should be using instead.
   IF place IS NULL THEN
     select coalesce(linked_place_id, place_id) as place_id,  country_code,
-           housenumber, postcode,
+           null::text as housenumber, postcode,
            class, type,
            null as name, address,
            null as centroid