adapt to release 3.6.0

.github/ISSUE_TEMPLATE/feature_request.md

@@ -7,16 +7,16 @@ assignees: ''
 
 ---
 
-<!-- Before opening a new feature request, please search through the open issue to check that your request hasn't been reported already. -->
+Before opening a new feature request, please search through the open issue to check that your request hasn't been reported already.
 
 **Is your feature request related to a problem? Please describe.**
-<!-- A clear and concise description of what the problem is. Ex. I'm always frustrated when [...] -->
+A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
 
 **Describe the solution you'd like**
-<!-- A clear and concise description of what you want to happen. -->
+A clear and concise description of what you want to happen.
 
 **Describe alternatives you've considered**
-<!-- A clear and concise description of any alternative solutions or features you've considered. -->
+A clear and concise description of any alternative solutions or features you've considered.
 
 **Additional context**
-<!-- Add any other context or screenshots about the feature request here. -->
+Add any other context or screenshots about the feature request here.

.github/ISSUE_TEMPLATE/report-issues-with-search-results.md

@@ -9,7 +9,7 @@ assignees: ''
 
 ## What did you search for?
 
-<!-- Please try to provide a link to your search. You  can go to https://nominatim.openstreetmap.org and repeat your search there. If you originally found the issue somewhere else, please tell us what software/website you were using. -->
+Please try to provide a link to your search. You  can go to https://nominatim.openstreetmap.org and repeat your search there. If you originally found the issue somewhere else, please tell us what software/website you were using.
 
 ## What result did you get?
 
@@ -17,11 +17,11 @@ assignees: ''
 
 **Is the result in the right place and just named wrongly?** 
 
-<!-- Please tell us the display name you expected. -->
+Please tell us the display name you expected.
 
 **Is the result missing completely?**
 
-<!-- Make sure that the data you are looking for is in OpenStreetMap. Provide a link to the OpenStreetMap object or if you cannot get it, a link to the map on https://openstreetmap.org where you expect the result to be.
+Make sure that the data you are looking for is in OpenStreetMap. Provide a link to the OpenStreetMap object or if you cannot get it, a link to the map on https://openstreetmap.org where you expect the result to be.
 
 To get the link to the OSM object, you can try the following:
 
@@ -30,8 +30,7 @@ To get the link to the OSM object, you can try the following:
  * Click on the question mark on the right side of the map. You get a question cursor. Use it to click on the map where your object is located.
  * Find the object of interest in the list that appears on the left side.
  * Click on the object and report back the URL that the browser shows.
--->
 
 ## Further details
 
-<!-- Anything else we should know about the search. Particularities with addresses in the area etc. -->
+Anything else we should know about the search. Particularities with addresses in the area etc.

.github/ISSUE_TEMPLATE/report-problems-with-the-software.md

@@ -7,13 +7,13 @@ assignees: ''
 
 ---
 
-<!-- Note: if you are installing Nominatim through a docker image, you should report issues with the installation process with the docker repository first. -->
+___Note: if you are installing Nominatim through a docker image, you should report issues with the installation process with the docker repository first.___
 
 **Describe the bug**
-<!-- A clear and concise description of what the bug is. -->
+A clear and concise description of what the bug is.
 
 **To Reproduce**
-<!-- Please describe what you did to get to the issue. -->
+Please describe what you did to get to the issue.
 
 **Software Environment (please complete the following information):**
 - Nominatim version: 
@@ -27,10 +27,5 @@ assignees: ''
 - type and size of disks:
 - bare metal/AWS/other cloud service: 
 
-**Postgresql Configuration:**
-
-<!-- List any configuration items you changed in your postgresql configuration. -->
-
 **Additional context**
-
-<!-- Add any other context about the problem here. -->
+Add any other context about the problem here.

.github/actions/build-nominatim/action.yml

@@ -4,26 +4,25 @@ runs:
     using: "composite"
 
     steps:
-        - name: Install prerequisites
-          run: |
-            sudo apt-get install -y -qq libboost-system-dev libboost-filesystem-dev libexpat1-dev zlib1g-dev libbz2-dev libpq-dev libproj-dev libicu-dev python3-psycopg2 python3-pyosmium python3-dotenv python3-psutil python3-jinja2 python3-icu python3-argparse-manpage
-          shell: bash
-
-        - name: Download dependencies
-          run: |
-              if [ ! -f country_grid.sql.gz ]; then
-                  wget --no-verbose https://www.nominatim.org/data/country_grid.sql.gz
-              fi
-              cp country_grid.sql.gz Nominatim/data/country_osm_grid.sql.gz
+        - name: Install prerequisits
+          run: sudo apt-get install -y -qq libboost-system-dev libboost-filesystem-dev libexpat1-dev zlib1g-dev libbz2-dev libpq-dev libproj-dev python3-psycopg2 python3-pyosmium
           shell: bash
 
         - name: Configure
-          run: mkdir build && cd build && cmake ../Nominatim
+          run: mkdir build && cd build && cmake ..
           shell: bash
 
         - name: Build
           run: |
               make -j2 all
-              sudo make install
+              ./utils/setup.php --setup-website
           shell: bash
           working-directory: build
+
+        - name: Download dependencies
+          run: |
+              if [ ! -f data/country_osm_grid.sql.gz ]; then
+                  wget --no-verbose -O data/country_osm_grid.sql.gz https://www.nominatim.org/data/country_grid.sql.gz
+              fi
+          shell: bash
+

.github/workflows/ci-tests.yml

@@ -19,13 +19,6 @@ jobs:
             - uses: actions/checkout@v2
               with:
                   submodules: true
-                  path: Nominatim
-
-            - name: Setup PHP
-              uses: shivammathur/setup-php@v2
-              with:
-                  php-version: '7.4'
-                  tools: phpunit, phpcs
 
             - name: Get Date
               id: get-date
@@ -36,37 +29,31 @@ jobs:
             - uses: actions/cache@v2
               with:
                   path: |
-                     country_grid.sql.gz
-                  key: nominatim-country-data-${{ steps.get-date.outputs.date }}
+                      data/country_osm_grid.sql.gz
+                      monaco-latest.osm.pbf
+                  key: nominatim-data-${{ steps.get-date.outputs.date }}
 
-            - uses: ./Nominatim/.github/actions/setup-postgresql
+            - uses: ./.github/actions/setup-postgresql
               with:
                   postgresql-version: ${{ matrix.postgresql }}
                   postgis-version: ${{ matrix.postgis }}
-            - uses: ./Nominatim/.github/actions/build-nominatim
+            - uses: ./.github/actions/build-nominatim
 
             - name: Install test prerequsites
-              run: sudo apt-get install -y -qq php-codesniffer pylint python3-pytest python3-behave
+              run: |
+                   sudo apt-get install -y -qq php-codesniffer python3-tidylib
+                   sudo pip3 install behave nose
 
             - name: PHP linting
               run: phpcs --report-width=120 .
-              working-directory: Nominatim
-
-            - name: Python linting
-              run: pylint --extension-pkg-whitelist=osmium nominatim
-              working-directory: Nominatim
 
             - name: PHP unit tests
               run: phpunit ./
-              working-directory: Nominatim/test/php
-
-            - name: Python unit tests
-              run: py.test-3 test/python
-              working-directory: Nominatim
+              working-directory: test/php
 
             - name: BDD tests
-              run: behave -DREMOVE_TEMPLATE=1 -DBUILDDIR=$GITHUB_WORKSPACE/build --format=progress3
-              working-directory: Nominatim/test/bdd
+              run: behave -DREMOVE_TEMPLATE=1 --format=progress3 db osm2pgsql
+              working-directory: test/bdd
 
     import:
         runs-on: ubuntu-20.04
@@ -75,7 +62,6 @@ jobs:
             - uses: actions/checkout@v2
               with:
                   submodules: true
-                  path: Nominatim
 
             - name: Get Date
               id: get-date
@@ -86,55 +72,49 @@ jobs:
             - uses: actions/cache@v2
               with:
                   path: |
-                     country_grid.sql.gz
-                  key: nominatim-country-data-${{ steps.get-date.outputs.date }}
+                      data/country_osm_grid.sql.gz
+                      monaco-latest.osm.pbf
+                  key: nominatim-data-${{ steps.get-date.outputs.date }}
 
-            - uses: actions/cache@v2
-              with:
-                  path: |
-                     monaco-latest.osm.pbf
-                  key: nominatim-test-data-${{ steps.get-date.outputs.date }}
-
-            - uses: ./Nominatim/.github/actions/setup-postgresql
+            - uses: ./.github/actions/setup-postgresql
               with:
                   postgresql-version: 13
                   postgis-version: 3
-            - uses: ./Nominatim/.github/actions/build-nominatim
+            - uses: ./.github/actions/build-nominatim
 
-            - name: Clean installation
-              run: rm -rf Nominatim build
-              shell: bash
+            - name: Create configuration
+              run: |
+                   echo '<?php' > settings/local.php
+                   echo " @define('CONST_Pyosmium_Binary', '/usr/lib/python3-pyosmium/pyosmium-get-changes');" >> settings/local.php
+              working-directory: build
 
-            - name: Prepare import environment
+            - name: Download import data
               run: |
                   if [ ! -f monaco-latest.osm.pbf ]; then
                       wget --no-verbose https://download.geofabrik.de/europe/monaco-latest.osm.pbf
                   fi
-                  mkdir data-env
-                  cd data-env
               shell: bash
 
             - name: Import
-              run: nominatim import --osm-file ../monaco-latest.osm.pbf
-              shell: bash
-              working-directory: data-env
+              run: php ./utils/setup.php --osm-file ../monaco-latest.osm.pbf --osm2pgsql-cache 500 --all
+              working-directory: build
 
             - name: Import special phrases
-              run: nominatim special-phrases --import-from-wiki
-              working-directory: data-env
+              run: php ./utils/specialphrases.php --wiki-import | psql -d nominatim
+              working-directory: build
 
             - name: Check import
-              run: nominatim admin --check-database
-              working-directory: data-env
+              run: php ./utils/check_import_finished.php
+              working-directory: build
 
             - name: Run update
               run: |
-                   nominatim replication --init
-                   nominatim replication --once
-              working-directory: data-env
+                   php ./utils/update.php --init-updates
+                   php ./utils/update.php --import-osmosis
+              working-directory: build
 
             - name: Run reverse-only import
-              run : nominatim import --osm-file ../monaco-latest.osm.pbf --reverse-only
-              working-directory: data-env
-              env:
-                  NOMINATIM_DATABASE_DSN: pgsql:dbname=reverse
+              run : |
+                  dropdb nominatim
+                  php ./utils/setup.php --osm-file ../monaco-latest.osm.pbf --reverse-only --all
+              working-directory: build

.gitignore

@@ -9,4 +9,3 @@ data/wiki_specialphrases.sql
 data/osmosischange.osc
 
 .vagrant
-data/country_osm_grid.sql.gz

.pylintrc

@@ -1,12 +0,0 @@
-[MASTER]
-
-extension-pkg-whitelist=osmium
-ignored-modules=icu
-
-[MESSAGES CONTROL]
-
-[TYPECHECK]
-
-# closing added here because it sometimes triggers a false positive with
-# 'with' statements.
-ignored-classes=NominatimArgs,closing

CMakeLists.txt

@@ -6,7 +6,7 @@
 #
 #-----------------------------------------------------------------------------
 
-cmake_minimum_required(VERSION 3.0 FATAL_ERROR)
+cmake_minimum_required(VERSION 2.8 FATAL_ERROR)
 list(APPEND CMAKE_MODULE_PATH "${CMAKE_SOURCE_DIR}/cmake")
 
 
@@ -19,7 +19,7 @@ list(APPEND CMAKE_MODULE_PATH "${CMAKE_SOURCE_DIR}/cmake")
 project(nominatim)
 
 set(NOMINATIM_VERSION_MAJOR 3)
-set(NOMINATIM_VERSION_MINOR 7)
+set(NOMINATIM_VERSION_MINOR 6)
 set(NOMINATIM_VERSION_PATCH 0)
 
 set(NOMINATIM_VERSION "${NOMINATIM_VERSION_MAJOR}.${NOMINATIM_VERSION_MINOR}.${NOMINATIM_VERSION_PATCH}")
@@ -36,7 +36,6 @@ set(BUILD_API on CACHE BOOL "Build everything for the API server")
 set(BUILD_MODULE on CACHE BOOL "Build PostgreSQL module")
 set(BUILD_TESTS on CACHE BOOL "Build test suite")
 set(BUILD_DOCS on CACHE BOOL "Build documentation")
-set(BUILD_MANPAGE on CACHE BOOL "Build Manual Page")
 set(BUILD_OSM2PGSQL on CACHE BOOL "Build osm2pgsql (expert only)")
 
 #-----------------------------------------------------------------------------
@@ -58,11 +57,20 @@ endif()
 
 
 #-----------------------------------------------------------------------------
-#  python (imports/updates only)
+#  python and pyosmium (imports/updates only)
 #-----------------------------------------------------------------------------
 
 if (BUILD_IMPORTER)
-    find_package(PythonInterp 3.5 REQUIRED)
+    find_package(PythonInterp 3)
+
+    find_program(PYOSMIUM pyosmium-get-changes)
+    if (NOT EXISTS "${PYOSMIUM}")
+        set(PYOSMIUM_PATH "")
+            message(WARNING "pyosmium-get-changes not found (required for updates)")
+    else()
+        set(PYOSMIUM_PATH "${PYOSMIUM}")
+        message(STATUS "Using pyosmium-get-changes at ${PYOSMIUM_PATH}")
+    endif()
 endif()
 
 #-----------------------------------------------------------------------------
@@ -78,19 +86,8 @@ if (BUILD_API OR BUILD_IMPORTER)
     # sanity check if PHP binary exists
     if (NOT EXISTS ${PHP_BIN})
         message(FATAL_ERROR "PHP binary not found. Install php or provide location with -DPHP_BIN=/path/php ")
-    else()
-        message (STATUS "Using PHP binary " ${PHP_BIN})
-    endif()
-    if (NOT PHPCGI_BIN)
-        find_program (PHPCGI_BIN php-cgi)
-    endif()
-    # sanity check if PHP binary exists
-    if (NOT EXISTS ${PHPCGI_BIN})
-        message(WARNING "php-cgi binary not found. nominatim tool will not provide query functions.")
-        set (PHPCGI_BIN "")
-    else()
-        message (STATUS "Using php-cgi binary " ${PHPCGI_BIN})
     endif()
+    message (STATUS "Using PHP binary " ${PHP_BIN})
 endif()
 
 #-----------------------------------------------------------------------------
@@ -98,36 +95,70 @@ endif()
 #-----------------------------------------------------------------------------
 
 if (BUILD_IMPORTER)
-   find_file(COUNTRY_GRID_FILE country_osm_grid.sql.gz
-             PATHS ${PROJECT_SOURCE_DIR}/data
-             NO_DEFAULT_PATH
-             DOC "Location of the country grid file."
-            )
-
-   if (NOT COUNTRY_GRID_FILE)
-       message(FATAL_ERROR "\nYou need to download the country_osm_grid first:\n"
-                           "    wget -O ${PROJECT_SOURCE_DIR}/data/country_osm_grid.sql.gz https://www.nominatim.org/data/country_grid.sql.gz")
-   endif()
-
    set(CUSTOMSCRIPTS
-       check_import_finished.php
-       country_languages.php
-       export.php
-       query.php
-       setup.php
-       update.php
-       warm.php
+       utils/check_import_finished.php
+       utils/country_languages.php
+       utils/importWikipedia.php
+       utils/export.php
+       utils/query.php
+       utils/setup.php
+       utils/specialphrases.php
+       utils/update.php
+       utils/warm.php
       )
 
    foreach (script_source ${CUSTOMSCRIPTS})
        configure_file(${PROJECT_SOURCE_DIR}/cmake/script.tmpl
-                      ${PROJECT_BINARY_DIR}/utils/${script_source})
+                      ${PROJECT_BINARY_DIR}/${script_source})
+   endforeach()
+endif()
+
+#-----------------------------------------------------------------------------
+# webserver scripts (API only)
+#-----------------------------------------------------------------------------
+
+if (BUILD_API)
+   set(WEBSITESCRIPTS
+       website/deletable.php
+       website/details.php
+       website/lookup.php
+       website/polygons.php
+       website/reverse.php
+       website/search.php
+       website/status.php
+   )
+
+   foreach (script_source ${WEBSITESCRIPTS})
+       configure_file(${PROJECT_SOURCE_DIR}/cmake/website.tmpl
+                      ${PROJECT_BINARY_DIR}/${script_source})
    endforeach()
 
-   configure_file(${PROJECT_SOURCE_DIR}/cmake/tool.tmpl
-                  ${PROJECT_BINARY_DIR}/nominatim)
+   set(WEBPATHS css images js)
+
+   foreach (wp ${WEBPATHS})
+       execute_process(
+           COMMAND ln -sf ${PROJECT_SOURCE_DIR}/website/${wp} ${PROJECT_BINARY_DIR}/website/
+       )
+   endforeach()
+
+   add_custom_target(serve
+           php -S 127.0.0.1:8088
+           WORKING_DIRECTORY ${PROJECT_BINARY_DIR}/website
+   )
+
+   add_custom_target(serve-global
+           php -S 0.0.0.0:8088
+           WORKING_DIRECTORY ${PROJECT_BINARY_DIR}/website
+   )
 endif()
 
+#-----------------------------------------------------------------------------
+# default settings
+#-----------------------------------------------------------------------------
+
+configure_file(${PROJECT_SOURCE_DIR}/settings/defaults.php
+               ${PROJECT_BINARY_DIR}/settings/settings.php)
+
 #-----------------------------------------------------------------------------
 # Tests
 #-----------------------------------------------------------------------------
@@ -137,60 +168,21 @@ if (BUILD_TESTS)
 
     set(TEST_BDD db osm2pgsql api)
 
-    find_program(PYTHON_BEHAVE behave)
-    find_program(PYLINT NAMES pylint3 pylint)
-    find_program(PYTEST NAMES pytest py.test-3 py.test)
-    find_program(PHPCS phpcs)
-    find_program(PHPUNIT phpunit)
+    foreach (test ${TEST_BDD})
+        add_test(NAME bdd_${test}
+                 COMMAND behave ${test}
+                 WORKING_DIRECTORY ${PROJECT_SOURCE_DIR}/test/bdd)
+        set_tests_properties(bdd_${test}
+            PROPERTIES ENVIRONMENT "NOMINATIM_DIR=${PROJECT_BINARY_DIR}")
+    endforeach()
 
-    if (PYTHON_BEHAVE)
-        message(STATUS "Using Python behave binary ${PYTHON_BEHAVE}")
-        foreach (test ${TEST_BDD})
-            add_test(NAME bdd_${test}
-                     COMMAND ${PYTHON_BEHAVE} ${test}
-                     WORKING_DIRECTORY ${PROJECT_SOURCE_DIR}/test/bdd)
-            set_tests_properties(bdd_${test}
-                PROPERTIES ENVIRONMENT "NOMINATIM_DIR=${PROJECT_BINARY_DIR}")
-        endforeach()
-    else()
-        message(WARNING "behave not found. BDD tests disabled." )
-    endif()
+    add_test(NAME php
+             COMMAND phpunit ./
+             WORKING_DIRECTORY ${PROJECT_SOURCE_DIR}/test/php)
 
-    if (PHPUNIT)
-        message(STATUS "Using phpunit binary ${PHPUNIT}")
-        add_test(NAME php
-                 COMMAND ${PHPUNIT} ./
-                 WORKING_DIRECTORY ${PROJECT_SOURCE_DIR}/test/php)
-    else()
-        message(WARNING "phpunit not found. PHP unit tests disabled." )
-    endif()
-
-    if (PHPCS)
-        message(STATUS "Using phpcs binary ${PHPCS}")
-        add_test(NAME phpcs
-                 COMMAND ${PHPCS} --report-width=120 --colors lib website utils
-                 WORKING_DIRECTORY ${PROJECT_SOURCE_DIR})
-    else()
-        message(WARNING "phpcs not found. PHP linting tests disabled." )
-    endif()
-
-    if (PYLINT)
-        message(STATUS "Using pylint binary ${PYLINT}")
-        add_test(NAME pylint
-                 COMMAND ${PYLINT} nominatim
-                 WORKING_DIRECTORY ${PROJECT_SOURCE_DIR})
-    else()
-        message(WARNING "pylint not found. Python linting tests disabled.")
-    endif()
-
-    if (PYTEST)
-        message(STATUS "Using pytest binary ${PYTEST}")
-        add_test(NAME pytest
-                 COMMAND ${PYTEST} test/python
-                 WORKING_DIRECTORY ${PROJECT_SOURCE_DIR})
-    else()
-        message(WARNING "pytest not found. Python tests disabled." )
-    endif()
+    add_test(NAME phpcs
+             COMMAND phpcs --report-width=120 --colors lib website utils
+             WORKING_DIRECTORY ${PROJECT_SOURCE_DIR})
 endif()
 
 #-----------------------------------------------------------------------------
@@ -208,69 +200,3 @@ endif()
 if (BUILD_DOCS)
    add_subdirectory(docs)
 endif()
-
-#-----------------------------------------------------------------------------
-# Manual page
-#-----------------------------------------------------------------------------
-
-if (BUILD_MANPAGE)
-   add_subdirectory(manual)
-endif()
-
-#-----------------------------------------------------------------------------
-# Installation
-#-----------------------------------------------------------------------------
-
-
-include(GNUInstallDirs)
-set(NOMINATIM_DATADIR ${CMAKE_INSTALL_FULL_DATADIR}/${PROJECT_NAME})
-set(NOMINATIM_LIBDIR ${CMAKE_INSTALL_FULL_LIBDIR}/${PROJECT_NAME})
-set(NOMINATIM_CONFIGDIR ${CMAKE_INSTALL_FULL_SYSCONFDIR}/${PROJECT_NAME})
-
-if (BUILD_IMPORTER)
-    configure_file(${PROJECT_SOURCE_DIR}/cmake/tool-installed.tmpl installed.bin)
-    install(PROGRAMS ${PROJECT_BINARY_DIR}/installed.bin
-            DESTINATION ${CMAKE_INSTALL_BINDIR}
-            RENAME nominatim)
-
-    install(DIRECTORY nominatim
-            DESTINATION ${NOMINATIM_LIBDIR}/lib-python
-            FILES_MATCHING PATTERN "*.py"
-            PATTERN __pycache__ EXCLUDE)
-    install(DIRECTORY lib-sql DESTINATION ${NOMINATIM_LIBDIR})
-
-    install(FILES data/country_name.sql
-                  ${COUNTRY_GRID_FILE}
-                  data/words.sql
-            DESTINATION ${NOMINATIM_DATADIR})
-endif()
-
-if (BUILD_OSM2PGSQL)
-    if (${CMAKE_VERSION} VERSION_LESS 3.13)
-        # Installation of subdirectory targets was only introduced in 3.13.
-        # So just copy the osm2pgsql file for older versions.
-        install(PROGRAMS ${PROJECT_BINARY_DIR}/osm2pgsql/osm2pgsql
-                DESTINATION ${NOMINATIM_LIBDIR})
-    else()
-        install(TARGETS osm2pgsql RUNTIME DESTINATION ${NOMINATIM_LIBDIR})
-    endif()
-endif()
-
-if (BUILD_MODULE)
-    install(PROGRAMS ${PROJECT_BINARY_DIR}/module/nominatim.so
-            DESTINATION ${NOMINATIM_LIBDIR}/module)
-endif()
-
-if (BUILD_API)
-    install(DIRECTORY lib-php DESTINATION ${NOMINATIM_LIBDIR})
-endif()
-
-install(FILES settings/env.defaults
-              settings/address-levels.json
-              settings/phrase-settings.json
-              settings/import-admin.style
-              settings/import-street.style
-              settings/import-address.style
-              settings/import-full.style
-              settings/import-extratags.style
-        DESTINATION ${NOMINATIM_CONFIGDIR})

CONTRIBUTING.md

@@ -49,18 +49,22 @@ are in process of consolidating the style. The following rules apply:
  * for PHP variables use CamelCase with a prefixing letter indicating the type
    (i - integer, f - float, a - array, s - string, o - object)
 
-The coding style is enforced with PHPCS and pylint. It can be tested with:
+The coding style is enforced with PHPCS and can be tested with:
 
 ```
-phpcs --report-width=120 --colors .
-pylint3 --extension-pkg-whitelist=osmium nominatim
+  phpcs --report-width=120 --colors .
 ```
 
 ## Testing
 
-Before submitting a pull request make sure that the tests pass:
+Before submitting a pull request make sure that the following tests pass:
 
 ```
-  cd build
-  make test
+  cd test/bdd
+  behave -DBUILDDIR=<builddir> db osm2pgsql
+```
+
+```
+  cd test/php
+  phpunit ./
 ```

ChangeLog

@@ -1,26 +1,3 @@
-3.7.0
-
- * switch to dotenv for configuration file
- * introduce 'make install' (reorganising most of the code)
- * introduce nominatim tool as replacement for various php scripts
- * introduce project directories and allow multiple installations from same build
- * clean up BDD tests: drop nose, reorganise step code
- * simplify test database for API BDD tests and autoinstall database
- * port most of the code for command-line tools to Python
-   (thanks to @darkshredder and @AntoJvlt)
- * add tests for all tooling
- * replace pyosmium-get-changes with custom internal implementation using
-   pyosmium
- * improve search for queries with housenumber and partial terms
- * add database versioning
- * use jinja2 for preprocessing SQL files
- * introduce automatic migrations
- * reverse fix preference of interpolations over housenumbers
- * parallelize indexing of postcodes
- * add non-key indexes to speed up housenumber + street searches
- * switch housenumber field in placex to save transliterated names
-
-
 3.6.0
 
  * add full support for searching by and displaying of addr:* tags

README.md

@@ -24,14 +24,15 @@ Installing and running Nominatim is something for experienced system
 administrators only who can do some trouble-shooting themselves. We are sorry,
 but we can not provide installation support. We are all doing this in our free
 time and there is just so much of that time to go around. Do not open issues in
-our bug tracker if you need help. Use the discussions forum
-or ask for help on [help.openstreetmap.org](https://help.openstreetmap.org/).**
+our bug tracker if you need help. You can ask questions on the mailing list
+(see below) or on [help.openstreetmap.org](https://help.openstreetmap.org/).**
 
 The latest stable release can be downloaded from https://nominatim.org.
 There you can also find [installation instructions for the release](https://nominatim.org/release-docs/latest/admin/Installation), as well as an extensive [Troubleshooting/FAQ section](https://nominatim.org/release-docs/latest/admin/Faq/).
 
-[Detailed installation instructions for current master](https://nominatim.org/release-docs/develop/admin/Installation)
-can be found at nominatim.org as well.
+Detailed installation instructions for the development version can be
+found at [nominatim.org](https://nominatim.org/release-docs/develop/admin/Installation)
+as well.
 
 A quick summary of the necessary steps:
 
@@ -41,15 +42,12 @@ A quick summary of the necessary steps:
         cd build
         cmake ..
         make
-        sudo make install
 
-2. Create a project directory, get OSM data and import:
+2. Get OSM data and import:
 
-        mkdir nominatim-project
-        cd nominatim-project
-        nominatim import --osm-file <your planet file>
+        ./build/utils/setup.php --osm-file <your planet file> --all
 
-3. Point your webserver to the nominatim-project/website directory.
+3. Point your webserver to the ./build/website directory.
 
 
 License
@@ -61,14 +59,13 @@ The source code is available under a GPLv2 license.
 Contributing
 ============
 
-Contributions, bugreport and pull requests are welcome.
-For details see [contribution guide](CONTRIBUTING.md).
+Contributions are welcome. For details see [contribution guide](CONTRIBUTING.md).
+
+Both bug reports and pull requests are welcome.
 
 
-Questions and help
-==================
+Mailing list
+============
 
-For questions, community help and discussions you can use the
-[Github discussions forum](https://github.com/osm-search/Nominatim/discussions)
-or join the
-[geocoding mailing list](https://lists.openstreetmap.org/listinfo/geocoding).
+For questions you can join the geocoding mailing list, see
+https://lists.openstreetmap.org/listinfo/geocoding

Vagrantfile

@@ -30,7 +30,7 @@ Vagrant.configure("2") do |config|
     lv.memory = 2048
     lv.nested = true
     if ENV['CHECKOUT'] != 'y' then
-      override.vm.synced_folder ".", "/home/vagrant/Nominatim", type: 'rsync'
+      override.vm.synced_folder ".", "/home/vagrant/Nominatim", type: 'nfs'
     end
   end
 

cmake/script.tmpl

@@ -1,14 +1,4 @@
 #!@PHP_BIN@ -Cq
 <?php
-require('@CMAKE_SOURCE_DIR@/lib-php/dotenv_loader.php');
-
-@define('CONST_Default_ModulePath', '@CMAKE_BINARY_DIR@/module');
-@define('CONST_Default_Osm2pgsql', '@CMAKE_BINARY_DIR@/osm2pgsql/osm2pgsql');
-@define('CONST_DataDir', '@CMAKE_SOURCE_DIR@/data');
-@define('CONST_SqlDir', '@CMAKE_SOURCE_DIR@/lib-sql');
-@define('CONST_ConfigDir', '@CMAKE_SOURCE_DIR@/settings');
-
-loadDotEnv();
-$_SERVER['NOMINATIM_NOMINATIM_TOOL'] = '@CMAKE_BINARY_DIR@/nominatim';
-
-require_once('@CMAKE_SOURCE_DIR@/lib-php/admin/@script_source@');
+require_once(dirname(dirname(__FILE__)).'/settings/settings.php');
+require_once(CONST_BasePath.'/@script_source@');

cmake/tool-installed.tmpl

@@ -1,17 +0,0 @@
-#!/usr/bin/env python3
-import sys
-import os
-
-sys.path.insert(1, '@NOMINATIM_LIBDIR@/lib-python')
-
-os.environ['NOMINATIM_NOMINATIM_TOOL'] = os.path.abspath(__file__)
-
-from nominatim import cli
-
-exit(cli.nominatim(module_dir='@NOMINATIM_LIBDIR@/module',
-                   osm2pgsql_path='@NOMINATIM_LIBDIR@/osm2pgsql',
-                   phplib_dir='@NOMINATIM_LIBDIR@/lib-php',
-                   sqllib_dir='@NOMINATIM_LIBDIR@/lib-sql',
-                   data_dir='@NOMINATIM_DATADIR@',
-                   config_dir='@NOMINATIM_CONFIGDIR@',
-                   phpcgi_path='@PHPCGI_BIN@'))

cmake/tool.tmpl

@@ -1,17 +0,0 @@
-#!/usr/bin/env python3
-import sys
-import os
-
-sys.path.insert(1, '@CMAKE_SOURCE_DIR@')
-
-os.environ['NOMINATIM_NOMINATIM_TOOL'] = os.path.abspath(__file__)
-
-from nominatim import cli
-
-exit(cli.nominatim(module_dir='@CMAKE_BINARY_DIR@/module',
-                   osm2pgsql_path='@CMAKE_BINARY_DIR@/osm2pgsql/osm2pgsql',
-                   phplib_dir='@CMAKE_SOURCE_DIR@/lib-php',
-                   sqllib_dir='@CMAKE_SOURCE_DIR@/lib-sql',
-                   data_dir='@CMAKE_SOURCE_DIR@/data',
-                   config_dir='@CMAKE_SOURCE_DIR@/settings',
-                   phpcgi_path='@PHPCGI_BIN@'))

cmake/website.tmpl

@@ -0,0 +1,5 @@
+<?php
+@define('CONST_Debug', (isset($_GET['debug']) && $_GET['debug']));
+require_once(dirname(dirname(__FILE__)).'/settings/settings-frontend.php');
+
+require_once(CONST_BasePath.'/@script_source@');

data/gb_postcode_table.sql

@@ -0,0 +1,26 @@
+-- This data contains Ordnance Survey data © Crown copyright and database right 2010. 
+-- Code-Point Open contains Royal Mail data © Royal Mail copyright and database right 2010.
+-- OS data may be used under the terms of the OS OpenData licence:
+-- http://www.ordnancesurvey.co.uk/oswebsite/opendata/licence/docs/licence.pdf
+
+SET statement_timeout = 0;
+SET client_encoding = 'UTF8';
+SET standard_conforming_strings = off;
+SET check_function_bodies = false;
+SET client_min_messages = warning;
+SET escape_string_warning = off;
+
+SET search_path = public, pg_catalog;
+
+SET default_tablespace = '';
+
+SET default_with_oids = false;
+
+CREATE TABLE gb_postcode (
+    id integer,
+    postcode character varying(9),
+    geometry geometry,
+    CONSTRAINT enforce_dims_geometry CHECK ((st_ndims(geometry) = 2)),
+    CONSTRAINT enforce_srid_geometry CHECK ((st_srid(geometry) = 4326))
+);
+

data/us_postcode_table.sql

@@ -0,0 +1,16 @@
+SET statement_timeout = 0;
+SET client_encoding = 'UTF8';
+SET check_function_bodies = false;
+SET client_min_messages = warning;
+
+SET search_path = public, pg_catalog;
+
+SET default_tablespace = '';
+
+SET default_with_oids = false;
+
+CREATE TABLE us_postcode (
+    postcode text,
+    x double precision,
+    y double precision
+);

docs/admin/Advanced-Installations.md

@@ -5,7 +5,6 @@ your Nominatim database. It is assumed that you have already successfully
 installed the Nominatim software itself, if not return to the 
 [installation page](Installation.md).
 
-
 ## Importing multiple regions
 
 To import multiple regions in your database, you need to configure and run `utils/import_multiple_regions.sh` file. This script will set up the update directory which has the following structure:
@@ -28,14 +27,6 @@ update
 
 The `sequence.state` files will contain the sequence ID, which will be used by pyosmium to get updates. The tmp folder is used for import dump.
 
-### Installing PHP prerequisites
-
-The scripts this section still use the old PHP utils. They require
-the [Symphony dotenv](https://symfony.com/doc/4.1/components/dotenv.html)
-parser to work. To install it on Ubuntu/Debian, run:
-
-    sudo apt install php-symfony-dotenv
-
 ### Configuring multiple regions
 
 The file `import_multiple_regions.sh` needs to be edited as per your requirement:
@@ -56,16 +47,15 @@ The file `import_multiple_regions.sh` needs to be edited as per your requirement
     
         BASEURL="https://download.geofabrik.de"
         DOWNCOUNTRYPOSTFIX="-latest.osm.pbf"
-
-### Setting up multiple regions
-
+ 
 !!! tip
-    If your database already exists and you want to add more countries,
-    replace the setting up part
+    If your database already exists and you want to add more countries, replace the setting up part
     `${SETUPFILE} --osm-file ${UPDATEDIR}/tmp/combined.osm.pbf --all 2>&1`
     with `${UPDATEFILE} --import-file ${UPDATEDIR}/tmp/combined.osm.pbf --index --index-instances N 2>&1`
     where N is the numbers of CPUs in your system.
 
+### Setting up multiple regions
+
 Run the following command from your Nominatim directory after configuring the file.
 
     bash ./utils/import_multiple_regions.sh
@@ -164,7 +154,7 @@ Make sure that the PostgreSQL server package is installed on the machine
 the PostgreSQL server itself.
 
 Download and compile Nominatim as per standard instructions. Once done, you find
-the normalization library in `build/module/nominatim.so`. Copy the file to
+the nomrmalization library in `build/module/nominatim.so`. Copy the file to
 the database server at a location where it is readable and executable by the
 PostgreSQL server process.
 
@@ -172,11 +162,11 @@ PostgreSQL server process.
 
 On the client side you now need to configure the import to point to the
 correct location of the library **on the database server**. Add the following
-line to your your `.env` file:
+line to your your `settings/local.php` file:
 
 ```php
-NOMINATIM_DATABASE_MODULE_PATH="<directory on the database server where nominatim.so resides>"
+@define('CONST_Database_Module_Path', '<directory on the database server where nominatim.so resides>');
 ```
 
-Now change the `NOMINATIM_DATABASE_DSN` to point to your remote server and continue
+Now change the `CONST_Database_DSN` to point to your remote server and continue
 to follow the [standard instructions for importing](/admin/Import).

docs/admin/Deployment.md

@@ -1,7 +1,7 @@
 # Deploying Nominatim
 
 The Nominatim API is implemented as a PHP application. The `website/` directory
-in the project directory contains the configured website. You can serve this
+in the build directory contains the configured website. You can serve this
 in a production environment with any web server that is capable to run
 PHP scripts.
 
@@ -13,11 +13,10 @@ to run a web service. Please refer to the documentation of
 for background information on configuring the services.
 
 !!! Note
-    Throughout this page, we assume that your Nominatim project directory is
-    located in `/srv/nominatim-project` and that you have installed Nominatim
-    using the default installation prefix `/usr/local`. If you have put it
-    somewhere else, you need to adjust the commands and configuration
-    accordingly.
+    Throughout this page, we assume that your Nominatim build directory is
+    located in `/srv/nominatim/build` and the source code in
+    `/srv/nominatim/Nominatim`. If you have put it somewhere else, you
+    need to adjust the commands and configuration accordingly.
 
     We further assume that your web server runs as user `www-data`. Older
     versions of CentOS may still use the user name `apache`. You also need
@@ -30,7 +29,7 @@ web server user. You can check that the permissions are correct by accessing
 on of the php files as the web server user:
 
 ``` sh
-sudo -u www-data head -n 1 /srv/nominatim-project/website/search.php
+sudo -u www-data head -n 1 /srv/nominatim/build/website/search.php
 ```
 
 If this shows a permission error, then you need to adapt the permissions of
@@ -41,11 +40,11 @@ web server access. At a minimum the following SELinux labelling should be done
 for Nominatim:
 
 ``` sh
-sudo semanage fcontext -a -t httpd_sys_content_t "/usr/local/nominatim/lib/lib-php(/.*)?"
-sudo semanage fcontext -a -t httpd_sys_content_t "/srv/nominatim-project/website(/.*)?"
-sudo semanage fcontext -a -t lib_t "/srv/nominatim-project/module/nominatim.so"
-sudo restorecon -R -v /usr/local/lib/nominatim
-sudo restorecon -R -v /srv/nominatim-project
+sudo semanage fcontext -a -t httpd_sys_content_t "/srv/nominatim/Nominatim/(website|lib|settings)(/.*)?"
+sudo semanage fcontext -a -t httpd_sys_content_t "/srv/nominatim/build/(website|settings)(/.*)?"
+sudo semanage fcontext -a -t lib_t "/srv/nominatim/build/module/nominatim.so"
+sudo restorecon -R -v /srv/nominatim/Nominatim
+sudo restorecon -R -v /srv/nominatim/build
 ```
 
 ## Nominatim with Apache
@@ -66,13 +65,13 @@ Make sure your Apache configuration contains the required permissions for the
 directory and create an alias:
 
 ``` apache
-<Directory "/srv/nominatim-project/website">
+<Directory "/srv/nominatim/build/website">
   Options FollowSymLinks MultiViews
   AddType text/html   .php
   DirectoryIndex search.php
   Require all granted
 </Directory>
-Alias /nominatim /srv/nominatim-project/website
+Alias /nominatim /srv/nominatim/build/website
 ```
 
 After making changes in the apache config you need to restart apache.
@@ -111,7 +110,7 @@ Tell nginx that php files are special and to fastcgi_pass to the php-fpm
 unix socket by adding the location definition to the default configuration.
 
 ``` nginx
-root /srv/nominatim-project/website;
+root /srv/nominatim/build/website;
 index search.php;
 location / {
     try_files $uri $uri/ @php;

docs/admin/Faq.md

@@ -16,7 +16,7 @@ was killed. If it looks like this:
 then you can resume with the following command:
 
 ```sh
-nominatim import --continue indexing
+./utils/setup.php --index --create-search-indices --create-country-names
 ```
 
 If the reported rank is 26 or higher, you can also safely add `--index-noanalyse`.
@@ -31,7 +31,7 @@ list for hints.
 If it happened during index creation you can try rerunning the step with
 
 ```sh
-nominatim import --continue indexing
+./utils/setup.php --create-search-indices --ignore-errors
 ```
 
 Otherwise it's best to start the full setup from the beginning.
@@ -93,7 +93,7 @@ on a non-managed machine.
 
 ### I see the error: "function transliteration(text) does not exist"
 
-Reinstall the nominatim functions with `nominatim refresh --functions`
+Reinstall the nominatim functions with `setup.php --create--functions`
 and check for any errors, e.g. a missing `nominatim.so` file.
 
 ### I see the error: "ERROR: mmap (remap) failed"
@@ -113,8 +113,7 @@ Double-check clang is installed. Instead of `make` try running `make CLANG=true`
 
 ### nominatim UPDATE failed: ERROR: buffer 179261 is not owned by resource owner Portal
 
-Several users [reported this](https://github.com/openstreetmap/Nominatim/issues/1168)
-during the initial import of the database. It's
+Several users [reported this](https://github.com/openstreetmap/Nominatim/issues/1168) during the initial import of the database. It's
 something PostgreSQL internal Nominatim doesn't control. And PostgreSQL forums
 suggest it's threading related but definitely some kind of crash of a process.
 Users reported either rebooting the server, different hardware or just trying
@@ -203,7 +202,7 @@ See the installation instructions for a full list of required packages.
 ### I forgot to delete the flatnodes file before starting an import.
 
 That's fine. For each import the flatnodes file get overwritten.
-See [https://help.openstreetmap.org/questions/52419/nominatim-flatnode-storage](https://help.openstreetmap.org/questions/52419/nominatim-flatnode-storage)
+See [https://help.openstreetmap.org/questions/52419/nominatim-flatnode-storage]()
 for more information.
 
 
@@ -212,3 +211,11 @@ for more information.
 ### Can I import negative OSM ids into Nominatim?
 
 See [this question of Stackoverflow](https://help.openstreetmap.org/questions/64662/nominatim-flatnode-with-negative-id).
+
+### Missing XML or text declaration
+
+The website might show: `XML Parsing Error: XML or text declaration not at start of entity Location.`
+
+Make sure there are no spaces at the beginning of your `settings/local.php` file.
+
+

docs/admin/Import.md

@@ -1,54 +1,22 @@
 # Importing the Database
 
 The following instructions explain how to create a Nominatim database
-from an OSM planet file. It is assumed that you have already successfully
-installed the Nominatim software itself and the `nominatim` tool can be found
-in your `PATH`. If this is not the case, return to the
-[installation page](Installation.md).
+from an OSM planet file and how to keep the database up to date. It
+is assumed that you have already successfully installed the Nominatim
+software itself, if not return to the [installation page](Installation.md).
 
-## Creating the project directory
+## Configuration setup in settings/local.php
 
-Before you start the import, you should create a project directory for your
-new database installation. This directory receives all data that is related
-to a single Nominatim setup: configuration, extra data, etc. Create a project
-directory apart from the Nominatim software and change into the directory:
+The Nominatim server can be customized via the file `settings/local.php`
+in the build directory. Note that this is a PHP file, so it must always
+start like this:
 
-```
-mkdir ~/nominatim-planet
-cd ~/nominatim-planet
-```
+    <?php
 
-In the following, we refer to the project directory as `$PROJECT_DIR`. To be
-able to copy&paste instructions, you can export the appropriate variable:
-
-```
-export PROJECT_DIR=~/nominatim-planet
-```
-
-The Nominatim tool assumes per default that the current working directory is
-the project directory but you may explicitly state a different directory using
-the `--project-dir` parameter. The following instructions assume that you run
-all commands from the project directory.
-
-!!! tip "Migration Tip"
-
-    Nominatim used to be run directly from the build directory until version 3.6.
-    Essentially, the build directory functioned as the project directory
-    for the database installation. This setup still works and can be useful for
-    development purposes. It is not recommended anymore for production setups.
-    Create a project directory that is separate from the Nominatim software.
-
-### Configuration setup in `.env`
-
-The Nominatim server can be customized via an `.env` configuration file in the
-project directory. This is a file in [dotenv](https://github.com/theskumar/python-dotenv)
-format which looks the same as variable settings in a standard shell environment.
-You can also set the same configuration via environment variables. All
-settings have a `NOMINATIM_` prefix to avoid conflicts with other environment
-variables.
+without any leading spaces.
 
 There are lots of configuration settings you can tweak. Have a look
-at `settings/env.default` for a full list. Most should have a sensible default.
+at `settings/default.php` for a full list. Most should have a sensible default.
 
 #### Flatnode files
 
@@ -56,9 +24,9 @@ If you plan to import a large dataset (e.g. Europe, North America, planet),
 you should also enable flatnode storage of node locations. With this
 setting enabled, node coordinates are stored in a simple file instead
 of the database. This will save you import time and disk storage.
-Add to your `.env`:
+Add to your `settings/local.php`:
 
-    NOMINATIM_FLATNODE_FILE="/path/to/flatnode.file"
+    @define('CONST_Osm2pgsql_Flatnode_File', '/path/to/flatnode.file');
 
 Replace the second part with a suitable path on your system and make sure
 the directory exists. There should be at least 75GB of free space.
@@ -70,9 +38,9 @@ the directory exists. There should be at least 75GB of free space.
 Wikipedia can be used as an optional auxiliary data source to help indicate
 the importance of OSM features. Nominatim will work without this information
 but it will improve the quality of the results if this is installed.
-This data is available as a binary download. Put it into your project directory:
+This data is available as a binary download:
 
-    cd $PROJECT_DIR
+    cd $NOMINATIM_SOURCE_DIR/data
     wget https://www.nominatim.org/data/wikimedia-importance.sql.gz
 
 The file is about 400MB and adds around 4GB to the Nominatim database.
@@ -80,16 +48,15 @@ The file is about 400MB and adds around 4GB to the Nominatim database.
 !!! tip
     If you forgot to download the wikipedia rankings, you can also add
     importances after the import. Download the files, then run
-    `nominatim refresh --wiki-data --importance`. Updating importances for
-    a planet can take a couple of hours.
+    `./utils/setup.php --import-wikipedia-articles`
+    and `./utils/update.php --recompute-importance`.
 
 ### Great Britain, USA postcodes
 
 Nominatim can use postcodes from an external source to improve searches that
-involve a GB or US postcode. This data can be optionally downloaded into the
-project directory:
+involve a GB or US postcode. This data can be optionally downloaded:
 
-    cd $PROJECT_DIR
+    cd $NOMINATIM_SOURCE_DIR/data
     wget https://www.nominatim.org/data/gb_postcode_data.sql.gz
     wget https://www.nominatim.org/data/us_postcode_data.sql.gz
 
@@ -119,14 +86,11 @@ that Nominatim cannot compute the areas for some administrative areas.
 About half of the data in Nominatim's database is not really used for serving
 the API. It is only there to allow the data to be updated from the latest
 changes from OSM. For many uses these dynamic updates are not really required.
-If you don't plan to apply updates, you can run the import with the
-`--no-updates` parameter. This will drop the dynamic part of the database as
-soon as it is not required anymore.
-
-You can also drop the dynamic part later using the following command:
+If you don't plan to apply updates, the dynamic part of the database can be
+safely dropped using the following command:
 
 ```
-nominatim freeze
+./utils/setup.php --drop
 ```
 
 Note that you still need to provide for sufficient disk space for the initial
@@ -160,7 +124,7 @@ import styles available which only read selected data:
   Like the full style but also adds most of the OSM tags into the extratags
   column.
 
-The style can be changed with the configuration `NOMINATIM_IMPORT_STYLE`.
+The style can be changed with the configuration `CONST_Import_Style`.
 
 To give you an idea of the impact of using the different styles, the table
 below gives rough estimates of the final database size after import of a
@@ -192,7 +156,7 @@ Download the data to import. Then issue the following command
 from the **build directory** to start the import:
 
 ```sh
-nominatim import --osm-file <data file> 2>&1 | tee setup.log
+./utils/setup.php --osm-file <data file> --all 2>&1 | tee setup.log
 ```
 
 ### Notes on full planet imports
@@ -228,19 +192,29 @@ MB. Make sure you leave enough RAM for PostgreSQL and osm2pgsql as mentioned
 above. If the system starts swapping or you are getting out-of-memory errors,
 reduce the cache size or even consider using a flatnode file.
 
-
-### Testing the installation
+### Verify the import
 
 Run this script to verify all required tables and indices got created successfully.
 
 ```sh
-nominatim admin --check-database
+./utils/check_import_finished.php
 ```
 
+### Setting up the website
+
+Run the following command to set up the configuration file for the API frontend
+`settings/settings-frontend.php`. These settings are used in website/*.php files.
+
+```sh
+./utils/setup.php --setup-website
+```
+!!! Note
+    This step is not necessary if you use `--all` option while setting up the DB.
+
 Now you can try out your installation by running:
 
 ```sh
-nominatim serve
+make serve
 ```
 
 This runs a small test server normally used for development. You can use it
@@ -258,7 +232,7 @@ planner to make the right decisions. Recomputing them can improve the performanc
 of forward geocoding in particular under high load. To recompute word counts run:
 
 ```sh
-nominatim refresh --word-counts
+./utils/update.php --recompute-word-counts
 ```
 
 This will take a couple of hours for a full planet installation. You can
@@ -268,9 +242,10 @@ running this function.
 
 If you want to be able to search for places by their type through
 [special key phrases](https://wiki.openstreetmap.org/wiki/Nominatim/Special_Phrases)
-you also need to import these key phrases like this:
+you also need to enable these key phrases like this:
 
-    nominatim special-phrases --import-from-wiki
+    ./utils/specialphrases.php --wiki-import > specialphrases.sql
+    psql -d nominatim -f specialphrases.sql
 
 Note that this command downloads the phrases from the wiki link above. You
 need internet access for the step.
@@ -283,22 +258,26 @@ address set to complement the OSM house number data in the US. You can add
 TIGER data to your own Nominatim instance by following these steps. The
 entire US adds about 10GB to your database.
 
-  1. Get preprocessed TIGER 2020 data:
+  1. Get preprocessed TIGER 2019 data and unpack it into the
+     data directory in your Nominatim sources:
 
-        cd $PROJECT_DIR
-        wget https://nominatim.org/data/tiger2020-nominatim-preprocessed.tar.gz
+        cd Nominatim/data
+        wget https://nominatim.org/data/tiger2019-nominatim-preprocessed.tar.gz
+        tar xf tiger2019-nominatim-preprocessed.tar.gz
 
   2. Import the data into your Nominatim database:
 
-        nominatim add-data --tiger-data tiger2020-nominatim-preprocessed.tar.gz
+        ./utils/setup.php --import-tiger-data
 
-  3. Enable use of the Tiger data in your `.env` by adding:
+  3. Enable use of the Tiger data in your `settings/local.php` by adding:
 
-        echo NOMINATIM_USE_US_TIGER_DATA=yes >> .env
+         @define('CONST_Use_US_Tiger_Data', true);
 
   4. Apply the new settings:
 
-        nominatim refresh --functions
+```sh
+    ./utils/setup.php --create-functions --enable-diff-updates --create-partition-functions
+```
 
 
 See the [developer's guide](../develop/data-sources.md#us-census-tiger) for more

docs/admin/Installation.md

@@ -30,29 +30,23 @@ For compiling:
   * [proj](https://proj.org/)
   * [bzip2](http://www.bzip.org/)
   * [zlib](https://www.zlib.net/)
-  * [ICU](http://site.icu-project.org/)
   * [Boost libraries](https://www.boost.org/), including system and filesystem
   * PostgreSQL client libraries
   * a recent C++ compiler (gcc 5+ or Clang 3.8+)
 
 For running Nominatim:
 
-  * [PostgreSQL](https://www.postgresql.org) (9.5+ will work, 11+ strongly recommended)
+  * [PostgreSQL](https://www.postgresql.org) (9.3+)
   * [PostGIS](https://postgis.net) (2.2+)
-  * [Python 3](https://www.python.org/) (3.5+)
-  * [Psycopg2](https://www.psycopg.org) (2.7+)
-  * [Python Dotenv](https://github.com/theskumar/python-dotenv)
-  * [psutil](https://github.com/giampaolo/psutil)
-  * [Jinja2](https://palletsprojects.com/p/jinja/)
-  * [PyICU](https://pypi.org/project/PyICU/)
+  * [Python 3](https://www.python.org/)
+  * [Psycopg2](https://www.psycopg.org)
   * [PHP](https://php.net) (7.0 or later)
   * PHP-pgsql
   * PHP-intl (bundled with PHP)
-  * PHP-cgi (for running queries from the command line)
 
 For running continuous updates:
 
-  * [pyosmium](https://osmcode.org/pyosmium/)
+  * [pyosmium](https://osmcode.org/pyosmium/) (with Python 3)
 
 For dependencies for running tests and building documentation, see
 the [Development section](../develop/Development-Environment.md).
@@ -148,16 +142,6 @@ build at the same level as the Nominatim source directory run:
 ```
 cmake ../Nominatim
 make
-sudo make install
-```
-
-Nominatim installs itself into `/usr/local` per default. To choose a different
-installation directory add `-DCMAKE_INSTALL_PREFIX=<install root>` to the
-cmake command. Make sure that the `bin` directory is available in your path
-in that case, e.g.
-
-```
-export PATH=<install root>/bin:$PATH
 ```
 
 Now continue with [importing the database](Import.md).

docs/admin/Migration.md

@@ -1,66 +1,10 @@
 # Database Migrations
 
-Since version 3.7.0 Nominatim offers automatic migrations. Please follow
-the following steps:
+This page describes database migrations necessary to update existing databases
+to newer versions of Nominatim.
 
-* stop any updates that are potentially running
-* update Nominatim to the newer version
-* go to your project directory and run `nominatim admin --migrate`
-* (optionally) restart updates
-
-Below you find additional migrations and hints about other structural and
-breaking changes. **Please read them before running the migration.**
-
-!!! note
-    If you are migrating from a version <3.6, then you still have to follow
-    the manual migration steps up to 3.6.
-
-## 3.6.0 -> 3.7.0
-
-### New format and name of configuration file
-
-The configuration for an import is now saved in a `.env` file in the project
-directory. This file follows the dotenv format. For more information, see
-the [installation chapter](Import.md#configuration-setup-in-env).
-
-To migrate to the new system, create a new project directory, add the `.env`
-file and port your custom configuration from `settings/local.php`. Most
-settings are named similar and only have received a `NOMINATIM_` prefix.
-Use the default settings in `settings/env.defaults` as a reference.
-
-### New location for data files
-
-External data files for Wikipedia importance, postcodes etc. are no longer
-expected to reside in the source tree by default. Instead they will be searched
-in the project directory. If you have an automated setup script you must
-either adapt the download location or explicitly set the location of the
-files to the old place in your `.env`.
-
-### Introducing `nominatim` command line tool
-
-The various php utilities have been replaced with a single `nominatim`
-command line tool. Make sure to adapt any scripts. There is no direct 1:1
-matching between the old utilities and the commands of nominatim CLI. The
-following list gives you a list of nominatim sub-commands that contain
-functionality of each script:
-
-* ./utils/setup.php: `import`, `freeze`, `refresh`
-* ./utils/update.php: `replication`, `add-data`, `index`, `refresh`
-* ./utils/specialphrases.php: `special-phrases`
-* ./utils/check_import_finished.php: `admin`
-* ./utils/warm.php: `admin`
-* ./utils/export.php: `export`
-
-Try `nominatim <command> --help` for more information about each subcommand.
-
-`./utils/query.php` no longer exists in its old form. `nominatim search`
-provides a replacement but returns different output.
-
-### Switch to normalized house numbers
-
-The housenumber column in the placex table uses now normalized version.
-The automatic migration step will convert the column but this may take a
-very long time. It is advisable to take the machine offline while doing that.
+SQL statements should be executed from the PostgreSQL commandline. Execute
+`psql nominatim` to enter command line mode.
 
 ## 3.5.0 -> 3.6.0
 

docs/admin/Setup-Nominatim-UI.md

@@ -10,11 +10,12 @@ installation. For more details, please also have a look at the
 
 ## Installing nominatim-ui
 
-We provide regular releases of nominatim-ui that contain the packaged website.
-They do not need any special installation. Just download, configure
-and run it. Grab the latest release from
-[nominatim-ui's Github release page](https://github.com/osm-search/nominatim-ui/releases)
-and unpack it. You can use `nominatim-ui-x.x.x.tar.gz` or `nominatim-ui-x.x.x.zip`.
+nominatim-ui does not need any special installation, just download, configure
+and run it.
+
+Clone the source from github:
+
+    git clone https://github.com/osm-search/nominatim-ui
 
 Copy the example configuration into the right place:
 

docs/admin/Update.md

@@ -1,10 +1,8 @@
 # Updating the Database
 
 There are many different ways to update your Nominatim database.
-The following section describes how to keep it up-to-date using
-an [online replication service for OpenStreetMap data](https://wiki.openstreetmap.org/wiki/Planet.osm/diffs)
-For a list of other methods to add or update data see the output of
-`nominatim add-data --help`.
+The following section describes how to keep it up-to-date with Pyosmium.
+For a list of other methods see the output of `./utils/update.php --help`.
 
 !!! important
     If you have configured a flatnode file for the import, then you
@@ -19,37 +17,50 @@ Run (as the same user who will later run the updates):
 pip3 install --user osmium
 ```
 
+Nominatim needs a tool called `pyosmium-get-changes` which comes with
+Pyosmium. You need to tell Nominatim where to find it. Add the
+following line to your `settings/local.php`:
+
+    @define('CONST_Pyosmium_Binary', '/home/user/.local/bin/pyosmium-get-changes');
+
+The path above is fine if you used the `--user` parameter with pip.
+Replace `user` with your user name.
+
 #### Setting up the update process
 
 Next the update needs to be initialised. By default Nominatim is configured
 to update using the global minutely diffs.
 
 If you want a different update source you will need to add some settings
-to `.env`. For example, to use the daily country extracts
+to `settings/local.php`. For example, to use the daily country extracts
 diffs for Ireland from Geofabrik add the following:
 
-    # base URL of the replication service
-    NOMINATIM_REPLICATION_URL="https://download.geofabrik.de/europe/ireland-and-northern-ireland-updates"
-    # How often upstream publishes diffs
-    NOMINATIM_REPLICATION_UPDATE_INTERVAL=86400
-    # How long to sleep if no update found yet
-    NOMINATIM_REPLICATION_RECHECK_INTERVAL=900
+    // base URL of the replication service
+    @define('CONST_Replication_Url', 'https://download.geofabrik.de/europe/ireland-and-northern-ireland-updates');
+    // How often upstream publishes diffs
+    @define('CONST_Replication_Update_Interval', '86400');
+    // How long to sleep if no update found yet
+    @define('CONST_Replication_Recheck_Interval', '900');
 
 To set up the update process now run the following command:
 
-    nominatim replication --init
+    ./utils/update.php --init-updates
 
 It outputs the date where updates will start. Recheck that this date is
 what you expect.
 
-The `replication --init` command needs to be rerun whenever the replication
-service is changed.
+The `--init-updates` command needs to be rerun whenever the replication service
+is changed.
 
 #### Updating Nominatim
 
 The following command will keep your database constantly up to date:
 
-    nominatim replication
+    ./utils/update.php --import-osmosis-all
+
+(Note that even though the old name "import-osmosis-all" has been kept for
+compatibility reasons, Osmosis is not required to run this - it uses pyosmium
+behind the scenes.)
 
 If you have imported multiple country extracts and want to keep them
 up-to-date, [Advanced installations section](Advanced-Installations.md) contains instructions 

docs/api/Faq.md

@@ -58,4 +58,4 @@ The [Overpass API](https://wiki.openstreetmap.org/wiki/Overpass_API) is more
 suited for these kinds of queries.
 
 That said if you installed your own Nominatim instance you can use the
-`nominatim export` PHP script as basis to return such lists.
+`/utils/export.php` PHP script as basis to return such lists.

docs/api/Reverse.md

@@ -162,7 +162,7 @@ This overrides the specified machine readable format. (Default: 0)
   "licence":"Data © OpenStreetMap contributors, ODbL 1.0. https:\/\/www.openstreetmap.org\/copyright",
   "osm_type":"way",
   "osm_id":"280940520",
-  "lat":"-34.4391708",
+"lat":"-34.4391708",
   "lon":"-58.7064573",
   "place_rank":"26",
   "category":"highway",

docs/api/Status.md

@@ -35,16 +35,10 @@ will return HTTP code 200 and a structure
   {
       "status": 0,
       "message": "OK",
-      "data_updated": "2020-05-04T14:47:00+00:00",
-      "software_version": "3.6.0-0",
-      "database_version": "3.6.0-0"
+      "data_updated": "2020-05-04T14:47:00+00:00"
   }
 ```
 
-The `software_version` field contains the version of Nominatim used to serve
-the API. The `database_version` field contains the version of the data format
-in the database.
-
 On error will also return HTTP status code 200 and a structure with error
 code and message, e.g.
 

docs/develop/Development-Environment.md

@@ -26,14 +26,12 @@ following packages should get you started:
 ## Prerequisites for testing and documentation
 
 The Nominatim test suite consists of behavioural tests (using behave) and
-unit tests (using PHPUnit for PHP code and pytest for Python code).
-It has the following additional requirements:
+unit tests (using PHPUnit). It has the following additional requirements:
 
 * [behave test framework](https://behave.readthedocs.io) >= 1.2.5
+* [nose](https://nose.readthedocs.io)
 * [phpunit](https://phpunit.de) >= 7.3
 * [PHP CodeSniffer](https://github.com/squizlabs/PHP_CodeSniffer)
-* [Pylint](https://pylint.org/) (2.6.0 is used for the CI)
-* [pytest](https://pytest.org)
 
 The documentation is built with mkdocs:
 
@@ -49,9 +47,9 @@ To install all necessary packages run:
 
 ```sh
 sudo apt install php-cgi phpunit php-codesniffer \
-                 python3-pip python3-setuptools python3-dev pylint
+                 python3-pip python3-setuptools python3-dev
 
-pip3 install --user behave mkdocs pytest
+pip3 install --user behave nose mkdocs
 ```
 
 The `mkdocs` executable will be located in `.local/bin`. You may have to add
@@ -80,15 +78,58 @@ echo 'export PATH=~/.config/composer/vendor/bin:$PATH' > ~/.profile
 
 ## Executing Tests
 
-All tests are located in the `/test` directory.
+All tests are located in the `\test` directory.
 
-To run all tests just go to the build directory and run make:
+### Preparing the test database
+
+Some of the behavioural test expect a test database to be present. You need at
+least 2GB RAM and 10GB disk space to create the database.
+
+First create a separate directory for the test DB and fetch the test planet
+data and the Tiger data for South Dakota:
+
+```
+mkdir testdb
+cd testdb
+wget https://www.nominatim.org/data/test/nominatim-api-testdata.pbf
+wget -O - https://nominatim.org/data/tiger2018-nominatim-preprocessed.tar.gz | tar xz --wildcards --no-anchored '46*'
+```
+
+Configure and build Nominatim in the usual way:
+
+```
+cmake $USERNAME/Nominatim
+make
+```
+
+Copy the test settings:
+
+```
+cp $USERNAME/Nominatim/test/testdb/local.php settings/
+```
+
+Inspect the file to check that all settings are correct for your local setup.
+
+Now you can import the test database:
+
+```
+dropdb --if-exists test_api_nominatim
+./utils/setup.php --all --osm-file nominatim-api-testdb.pbf 2>&1 | tee import.log
+./utils/specialphrases.php --wiki-import | psql -d test_api_nominatim 2>&1 | tee -a import.log
+./utils/setup.php --import-tiger-data 2>&1 | tee -a import.log
+```
+
+### Running the tests
+
+To run all tests just go to the test directory and run make:
 
 ```sh
-cd build
-make test
+cd test
+make
 ```
 
+To skip tests that require the test database, run `make no-test-db` instead.
+
 For more information about the structure of the tests and how to change and
 extend the test suite, see the [Testing chapter](Testing.md).
 

docs/develop/Import.md

@@ -29,7 +29,7 @@ once with `class` of `highway` and once with a `class` of `bridge`. Thus the
 ## Configuring the Import
 
 How tags are interpreted and assigned to the different `place` columns can be
-configured via the import style configuration file (`NOMINATIM_IMPORT_STYLE`). This
+configured via the import style configuration file (`CONST_Import_style`). This
 is a JSON file which contains a list of rules which are matched against every
 tag of every object and then assign the tag its specific role.
 

docs/develop/Postcodes.md

@@ -14,7 +14,7 @@ country's format, e.g. if Swiss postcodes are 4 digits.
 ## Regular updating calculated postcodes
 
 The script to rerun the calculation is
-`nominatim refresh --postcodes`
+`build/utils/update.php --calculate-postcodes`
 and runs once per night on nominatim.openstreetmap.org.
 
 

docs/develop/Ranking.md

@@ -87,9 +87,9 @@ into the database. There are a few hard-coded rules for the assignment:
     * highway nodes
     * landuse that is not an area
 
-Other than that, the ranks can be freely assigned via the JSON file according
-to their type and the country they are in. The name of the config file to be
-used can be changed with the setting `NOMINATIM_ADDRESS_LEVEL_CONFIG`.
+Other than that, the ranks can be freely assigned via the JSON file
+defined with `CONST_Address_Level_Config` according to their type and
+the country they are in.
 
 The address level configuration must consist of an array of configuration
 entries, each containing a tag definition and an optional country array:

docs/develop/Testing.md

@@ -21,15 +21,14 @@ This test directory is sturctured as follows:
   | +-  api        Tests for API endpoints (search, reverse, etc.)
   |
   +-   php         PHP unit tests
-  +-   python      Python unit tests
   +-   scenes      Geometry test data
   +-   testdb      Base data for generating API test database
 ```
 
 ## PHP Unit Tests (`test/php`)
 
-Unit tests for PHP code can be found in the `php/` directory. They test selected
-PHP functions. Very low coverage.
+Unit tests can be found in the php/ directory. They test selected php functions.
+Very low coverage.
 
 To execute the test suite run
 
@@ -37,26 +36,11 @@ To execute the test suite run
     UNIT_TEST_DSN='pgsql:dbname=nominatim_unit_tests' phpunit ../
 
 It will read phpunit.xml which points to the library, test path, bootstrap
-strip and sets other parameters.
+strip and set other parameters.
 
 It will use (and destroy) a local database 'nominatim_unit_tests'. You can set
 a different connection string with e.g. UNIT_TEST_DSN='pgsql:dbname=foo_unit_tests'.
 
-## Python Unit Tests (`test/python`)
-
-Unit tests for Python code can be found in the `python/` directory. The goal is
-to have complete coverage of the Python library in `nominatim`.
-
-To execute the tests run
-
-    py.test-3 test/python
-
-or
-
-    pytest test/python
-
-The name of the pytest binary depends on your installation.
-
 ## BDD Functional Tests (`test/bdd`)
 
 Functional tests are written as BDD instructions. For more information on
@@ -83,17 +67,17 @@ The tests can be configured with a set of environment variables (`behave -D key=
                    the test databases (db tests)
  * `TEST_DB` - name of test database (db tests)
  * `API_TEST_DB` - name of the database containing the API test data (api tests)
- * `API_TEST_FILE` - OSM file to be imported into the API test database (api tests)
  * `DB_HOST` - (optional) hostname of database host
  * `DB_PORT` - (optional) port of database on host
  * `DB_USER` - (optional) username of database login
  * `DB_PASS` - (optional) password for database login
  * `SERVER_MODULE_PATH` - (optional) path on the Postgres server to Nominatim
                           module shared library file
- * `REMOVE_TEMPLATE` - if true, the template and API database will not be reused
-                       during the next run. Reusing the base templates speeds
-                       up tests considerably but might lead to outdated errors
-                       for some changes in the database layout.
+ * `TEST_SETTINGS_TEMPLATE` - file to write temporary Nominatim settings to
+ * `REMOVE_TEMPLATE` - if true, the template database will not be reused during
+                       the next run. Reusing the base templates speeds up tests
+                       considerably but might lead to outdated errors for some
+                       changes in the database layout.
  * `KEEP_TEST_DB` - if true, the test database will not be dropped after a test
                     is finished. Should only be used if one single scenario is
                     run, otherwise the result is undefined.
@@ -105,20 +89,23 @@ feature of behave which comes in handy when writing new tests.
 ### API Tests (`test/bdd/api`)
 
 These tests are meant to test the different API endpoints and their parameters.
-They require to import several datasets into a test database. This is normally
-done automatically during setup of the test. The API test database is then
-kept around and reused in subsequent runs of behave. Use `behave -DREMOVE_TEMPLATE`
-to force a reimport of the database.
+They require to import several datasets into a test database.
+See the [Development Setup chapter](Development-Environment.md#preparing-the-test-database)
+for instructions on how to set up this database.
 
-The official test dataset is saved in the file `test/testdb/apidb-test-data.pbf`
-and compromises the following data:
+The official test dataset was derived from the 180924 planet (note: such
+file no longer exists at https://planet.openstreetmap.org/planet/2018/).
+Newer planets are likely to work as well but you may see isolated test
+failures where the data has changed.
 
- * Geofabrik extract of Liechtenstein
- * extract of Autauga country, Alabama, US (for tests against Tiger data)
- * additional data from `test/testdb/additional_api_test.data.osm`
+The official test dataset can always be downloaded from
+[nominatim.org](https://www.nominatim.org/data/test/nominatim-api-testdata.pbf)
+To recreate the input data for the test database run:
 
-API tests should only be testing the functionality of the website PHP code.
-Most tests should be formulated as BDD DB creation tests (see below) instead.
+```
+wget https://ftp5.gwdg.de/pub/misc/openstreetmap/planet.openstreetmap.org/pbf/planet-180924.osm.pbf
+osmconvert planet-180924.osm.pbf -B=test/testdb/testdb.polys -o=testdb.pbf
+```
 
 #### Code Coverage
 
@@ -153,7 +140,3 @@ needs superuser rights for postgres.
 
 These tests check that data is imported correctly into the place table. They
 use the same template database as the DB Creation tests, so the same remarks apply.
-
-Note that most testing of the gazetteer output of osm2pgsql is done in the tests
-of osm2pgsql itself. The BDD tests are just there to ensure compatibility of
-the osm2pgsql and Nominatim code.

docs/mkdocs.yml

@@ -1,4 +1,4 @@
-site_name: Nominatim 3.7.2
+site_name: Nominatim 3.6.0
 theme: readthedocs
 docs_dir: ${CMAKE_CURRENT_BINARY_DIR}
 site_url: https://nominatim.org

lib-php/admin/check_import_finished.php

@@ -1,10 +0,0 @@
-<?php
-@define('CONST_LibDir', dirname(dirname(__FILE__)));
-
-require_once(CONST_LibDir.'/init-cmd.php');
-
-loadSettings(getcwd());
-
-(new \Nominatim\Shell(getSetting('NOMINATIM_TOOL')))
-    ->addParams('admin', '--check-database')
-    ->run();

lib-php/admin/specialphrases.php

@@ -1,11 +0,0 @@
-
-<?php
-@define('CONST_LibDir', dirname(dirname(__FILE__)));
-
-require_once(CONST_LibDir.'/init-cmd.php');
-
-loadSettings(getcwd());
-
-(new \Nominatim\Shell(getSetting('NOMINATIM_TOOL')))
-    ->addParams('special-phrases', '--import-from-wiki')
-    ->run();

lib-php/admin/update.php

@@ -1,236 +0,0 @@
-<?php
-@define('CONST_LibDir', dirname(dirname(__FILE__)));
-
-require_once(CONST_LibDir.'/init-cmd.php');
-require_once(CONST_LibDir.'/setup_functions.php');
-require_once(CONST_LibDir.'/setup/SetupClass.php');
-
-ini_set('memory_limit', '800M');
-
-use Nominatim\Setup\SetupFunctions as SetupFunctions;
-
-// (long-opt, short-opt, min-occurs, max-occurs, num-arguments, num-arguments, type, help)
-$aCMDOptions
-= array(
-   'Import / update / index osm data',
-   array('help', 'h', 0, 1, 0, 0, false, 'Show Help'),
-   array('quiet', 'q', 0, 1, 0, 0, 'bool', 'Quiet output'),
-   array('verbose', 'v', 0, 1, 0, 0, 'bool', 'Verbose output'),
-
-   array('init-updates', '', 0, 1, 0, 0, 'bool', 'Set up database for updating'),
-   array('check-for-updates', '', 0, 1, 0, 0, 'bool', 'Check if new updates are available'),
-   array('no-update-functions', '', 0, 1, 0, 0, 'bool', 'Do not update trigger functions to support differential updates (assuming the diff update logic is already present)'),
-   array('import-osmosis', '', 0, 1, 0, 0, 'bool', 'Import updates once'),
-   array('import-osmosis-all', '', 0, 1, 0, 0, 'bool', 'Import updates forever'),
-   array('no-index', '', 0, 1, 0, 0, 'bool', 'Do not index the new data'),
-
-   array('calculate-postcodes', '', 0, 1, 0, 0, 'bool', 'Update postcode centroid table'),
-
-   array('import-file', '', 0, 1, 1, 1, 'realpath', 'Re-import data from an OSM file'),
-   array('import-diff', '', 0, 1, 1, 1, 'realpath', 'Import a diff (osc) file from local file system'),
-   array('osm2pgsql-cache', '', 0, 1, 1, 1, 'int', 'Cache size used by osm2pgsql'),
-
-   array('import-node', '', 0, 1, 1, 1, 'int', 'Re-import node'),
-   array('import-way', '', 0, 1, 1, 1, 'int', 'Re-import way'),
-   array('import-relation', '', 0, 1, 1, 1, 'int', 'Re-import relation'),
-   array('import-from-main-api', '', 0, 1, 0, 0, 'bool', 'Use OSM API instead of Overpass to download objects'),
-
-   array('index', '', 0, 1, 0, 0, 'bool', 'Index'),
-   array('index-rank', '', 0, 1, 1, 1, 'int', 'Rank to start indexing from'),
-   array('index-instances', '', 0, 1, 1, 1, 'int', 'Number of indexing instances (threads)'),
-
-   array('recompute-word-counts', '', 0, 1, 0, 0, 'bool', 'Compute frequency of full-word search terms'),
-   array('update-address-levels', '', 0, 1, 0, 0, 'bool', 'Reimport address level configuration (EXPERT)'),
-   array('recompute-importance', '', 0, 1, 0, 0, 'bool', 'Recompute place importances'),
-
-   array('project-dir', '', 0, 1, 1, 1, 'realpath', 'Base directory of the Nominatim installation (default: .)'),
-  );
-
-getCmdOpt($_SERVER['argv'], $aCMDOptions, $aResult, true, true);
-
-loadSettings($aCMDResult['project-dir'] ?? getcwd());
-setupHTTPProxy();
-
-if (!isset($aResult['index-instances'])) $aResult['index-instances'] = 1;
-if (!isset($aResult['index-rank'])) $aResult['index-rank'] = 0;
-
-date_default_timezone_set('Etc/UTC');
-
-$oDB = new Nominatim\DB();
-$oDB->connect();
-$fPostgresVersion = $oDB->getPostgresVersion();
-
-$aDSNInfo = Nominatim\DB::parseDSN(getSetting('DATABASE_DSN'));
-if (!isset($aDSNInfo['port']) || !$aDSNInfo['port']) $aDSNInfo['port'] = 5432;
-
-// cache memory to be used by osm2pgsql, should not be more than the available memory
-$iCacheMemory = (isset($aResult['osm2pgsql-cache'])?$aResult['osm2pgsql-cache']:2000);
-if ($iCacheMemory + 500 > getTotalMemoryMB()) {
-    $iCacheMemory = getCacheMemoryMB();
-    echo "WARNING: resetting cache memory to $iCacheMemory\n";
-}
-
-$oOsm2pgsqlCmd = (new \Nominatim\Shell(getOsm2pgsqlBinary()))
-                 ->addParams('--hstore')
-                 ->addParams('--latlong')
-                 ->addParams('--append')
-                 ->addParams('--slim')
-                 ->addParams('--with-forward-dependencies', 'false')
-                 ->addParams('--log-progress', 'true')
-                 ->addParams('--number-processes', 1)
-                 ->addParams('--cache', $iCacheMemory)
-                 ->addParams('--output', 'gazetteer')
-                 ->addParams('--style', getImportStyle())
-                 ->addParams('--database', $aDSNInfo['database'])
-                 ->addParams('--port', $aDSNInfo['port']);
-
-if (isset($aDSNInfo['hostspec']) && $aDSNInfo['hostspec']) {
-    $oOsm2pgsqlCmd->addParams('--host', $aDSNInfo['hostspec']);
-}
-if (isset($aDSNInfo['username']) && $aDSNInfo['username']) {
-    $oOsm2pgsqlCmd->addParams('--user', $aDSNInfo['username']);
-}
-if (isset($aDSNInfo['password']) && $aDSNInfo['password']) {
-    $oOsm2pgsqlCmd->addEnvPair('PGPASSWORD', $aDSNInfo['password']);
-}
-if (getSetting('FLATNODE_FILE')) {
-    $oOsm2pgsqlCmd->addParams('--flat-nodes', getSetting('FLATNODE_FILE'));
-}
-if ($fPostgresVersion >= 11.0) {
-    $oOsm2pgsqlCmd->addEnvPair(
-        'PGOPTIONS',
-        '-c jit=off -c max_parallel_workers_per_gather=0'
-    );
-}
-
-$oNominatimCmd = new \Nominatim\Shell(getSetting('NOMINATIM_TOOL'));
-
-function run($oCmd)
-{
-    global $aCMDResult;
-    if ($aCMDResult['quiet'] ?? false) {
-        $oCmd->addParams('--quiet');
-    }
-    if ($aCMDResult['verbose'] ?? false) {
-        $oCmd->addParams('--verbose');
-    }
-    $oCmd->run(true);
-}
-
-
-if ($aResult['init-updates']) {
-    $oCmd = (clone($oNominatimCmd))->addParams('replication', '--init');
-
-    if ($aResult['no-update-functions']) {
-        $oCmd->addParams('--no-update-functions');
-    }
-
-    run($oCmd);
-}
-
-if ($aResult['check-for-updates']) {
-    exit((clone($oNominatimCmd))->addParams('replication', '--check-for-updates')->run());
-}
-
-if (isset($aResult['import-diff']) || isset($aResult['import-file'])) {
-    // import diffs and files directly (e.g. from osmosis --rri)
-    $sNextFile = isset($aResult['import-diff']) ? $aResult['import-diff'] : $aResult['import-file'];
-
-    if (!file_exists($sNextFile)) {
-        fail("Cannot open $sNextFile\n");
-    }
-
-    // Import the file
-    $oCMD = (clone $oOsm2pgsqlCmd)->addParams($sNextFile);
-    echo $oCMD->escapedCmd()."\n";
-    $iRet = $oCMD->run();
-
-    if ($iRet) {
-        fail("Error from osm2pgsql, $iRet\n");
-    }
-
-    // Don't update the import status - we don't know what this file contains
-}
-
-if ($aResult['calculate-postcodes']) {
-    run((clone($oNominatimCmd))->addParams('refresh', '--postcodes'));
-}
-
-$sTemporaryFile = CONST_InstallDir.'/osmosischange.osc';
-$bHaveDiff = false;
-$bUseOSMApi = isset($aResult['import-from-main-api']) && $aResult['import-from-main-api'];
-$sContentURL = '';
-if (isset($aResult['import-node']) && $aResult['import-node']) {
-    if ($bUseOSMApi) {
-        $sContentURL = 'https://www.openstreetmap.org/api/0.6/node/'.$aResult['import-node'];
-    } else {
-        $sContentURL = 'https://overpass-api.de/api/interpreter?data=node('.$aResult['import-node'].');out%20meta;';
-    }
-}
-
-if (isset($aResult['import-way']) && $aResult['import-way']) {
-    if ($bUseOSMApi) {
-        $sContentURL = 'https://www.openstreetmap.org/api/0.6/way/'.$aResult['import-way'].'/full';
-    } else {
-        $sContentURL = 'https://overpass-api.de/api/interpreter?data=(way('.$aResult['import-way'].');%3E;);out%20meta;';
-    }
-}
-
-if (isset($aResult['import-relation']) && $aResult['import-relation']) {
-    if ($bUseOSMApi) {
-        $sContentURL = 'https://www.openstreetmap.org/api/0.6/relation/'.$aResult['import-relation'].'/full';
-    } else {
-        $sContentURL = 'https://overpass-api.de/api/interpreter?data=(rel(id:'.$aResult['import-relation'].');%3E;);out%20meta;';
-    }
-}
-
-if ($sContentURL) {
-    file_put_contents($sTemporaryFile, file_get_contents($sContentURL));
-    $bHaveDiff = true;
-}
-
-if ($bHaveDiff) {
-    // import generated change file
-
-    $oCMD = (clone $oOsm2pgsqlCmd)->addParams($sTemporaryFile);
-    echo $oCMD->escapedCmd()."\n";
-
-    $iRet = $oCMD->run();
-    if ($iRet) {
-        fail("osm2pgsql exited with error level $iRet\n");
-    }
-}
-
-if ($aResult['recompute-word-counts']) {
-    run((clone($oNominatimCmd))->addParams('refresh', '--word-counts'));
-}
-
-if ($aResult['index']) {
-    run((clone $oNominatimCmd)
-        ->addParams('index', '--minrank', $aResult['index-rank'])
-        ->addParams('--threads', $aResult['index-instances']));
-}
-
-if ($aResult['update-address-levels']) {
-    run((clone($oNominatimCmd))->addParams('refresh', '--address-levels'));
-}
-
-if ($aResult['recompute-importance']) {
-    run((clone($oNominatimCmd))->addParams('refresh', '--importance'));
-}
-
-if ($aResult['import-osmosis'] || $aResult['import-osmosis-all']) {
-    $oCmd = (clone($oNominatimCmd))
-              ->addParams('replication')
-              ->addParams('--threads', $aResult['index-instances']);
-
-    if (!$aResult['import-osmosis-all']) {
-        $oCmd->addParams('--once');
-    }
-
-    if ($aResult['no-index']) {
-        $oCmd->addParams('--no-index');
-    }
-
-    run($oCmd);
-}

lib-php/dotenv_loader.php

@@ -1,13 +0,0 @@
-<?php
-
-require('Symfony/Component/Dotenv/autoload.php');
-
-function loadDotEnv()
-{
-    $dotenv = new \Symfony\Component\Dotenv\Dotenv();
-    $dotenv->load(CONST_ConfigDir.'/env.defaults');
-
-    if (file_exists('.env')) {
-        $dotenv->load('.env');
-    }
-}

lib-php/init-cmd.php

@@ -1,5 +0,0 @@
-<?php
-
-require_once('init.php');
-require_once('cmd.php');
-require_once('DebugNone.php');

lib-php/init.php

@@ -1,4 +0,0 @@
-<?php
-
-require_once(CONST_LibDir.'/lib.php');
-require_once(CONST_LibDir.'/DB.php');

lib-php/migration/PhraseSettingsToJson.php

@@ -1,19 +0,0 @@
-<?php
-
-$phpPhraseSettingsFile = $argv[1];
-$jsonPhraseSettingsFile = dirname($phpPhraseSettingsFile).'/'.basename($phpPhraseSettingsFile, '.php').'.json';
-
-if (file_exists($phpPhraseSettingsFile) && !file_exists($jsonPhraseSettingsFile)) {
-    include $phpPhraseSettingsFile;
-
-    $data = array();
-
-    if (isset($aTagsBlacklist))
-        $data['blackList'] = $aTagsBlacklist;
-    if (isset($aTagsWhitelist))
-        $data['whiteList'] = $aTagsWhitelist;
-
-    $jsonFile = fopen($jsonPhraseSettingsFile, 'w');
-    fwrite($jsonFile, json_encode($data));
-    fclose($jsonFile);
-}

lib-php/output.php

@@ -1,18 +0,0 @@
-<?php
-
-
-function formatOSMType($sType, $bIncludeExternal = true)
-{
-    if ($sType == 'N') return 'node';
-    if ($sType == 'W') return 'way';
-    if ($sType == 'R') return 'relation';
-
-    if (!$bIncludeExternal) return '';
-
-    if ($sType == 'T') return 'way';
-    if ($sType == 'I') return 'way';
-
-    // not handled: P, L
-
-    return '';
-}

lib-php/setup/SetupClass.php

@@ -1,261 +0,0 @@
-<?php
-
-namespace Nominatim\Setup;
-
-require_once(CONST_LibDir.'/Shell.php');
-
-class SetupFunctions
-{
-    protected $iInstances;
-    protected $aDSNInfo;
-    protected $bQuiet;
-    protected $bVerbose;
-    protected $sIgnoreErrors;
-    protected $bEnableDiffUpdates;
-    protected $bEnableDebugStatements;
-    protected $bDrop;
-    protected $oDB = null;
-    protected $oNominatimCmd;
-
-    public function __construct(array $aCMDResult)
-    {
-        // by default, use all but one processor, but never more than 15.
-        $this->iInstances = isset($aCMDResult['threads'])
-            ? $aCMDResult['threads']
-            : (min(16, getProcessorCount()) - 1);
-
-        if ($this->iInstances < 1) {
-            $this->iInstances = 1;
-            warn('resetting threads to '.$this->iInstances);
-        }
-
-        // parse database string
-        $this->aDSNInfo = \Nominatim\DB::parseDSN(getSetting('DATABASE_DSN'));
-        if (!isset($this->aDSNInfo['port'])) {
-            $this->aDSNInfo['port'] = 5432;
-        }
-
-        // setting member variables based on command line options stored in $aCMDResult
-        $this->bQuiet = isset($aCMDResult['quiet']) && $aCMDResult['quiet'];
-        $this->bVerbose = $aCMDResult['verbose'];
-
-        //setting default values which are not set by the update.php array
-        if (isset($aCMDResult['ignore-errors'])) {
-            $this->sIgnoreErrors = $aCMDResult['ignore-errors'];
-        } else {
-            $this->sIgnoreErrors = false;
-        }
-        if (isset($aCMDResult['enable-debug-statements'])) {
-            $this->bEnableDebugStatements = $aCMDResult['enable-debug-statements'];
-        } else {
-            $this->bEnableDebugStatements = false;
-        }
-        if (isset($aCMDResult['enable-diff-updates'])) {
-            $this->bEnableDiffUpdates = $aCMDResult['enable-diff-updates'];
-        } else {
-            $this->bEnableDiffUpdates = false;
-        }
-
-        $this->bDrop = isset($aCMDResult['drop']) && $aCMDResult['drop'];
-
-        $this->oNominatimCmd = new \Nominatim\Shell(getSetting('NOMINATIM_TOOL'));
-        if ($this->bQuiet) {
-            $this->oNominatimCmd->addParams('--quiet');
-        }
-        if ($this->bVerbose) {
-            $this->oNominatimCmd->addParams('--verbose');
-        }
-    }
-
-    public function calculatePostcodes($bCMDResultAll)
-    {
-        info('Calculate Postcodes');
-        $this->pgsqlRunScriptFile(CONST_SqlDir.'/postcode_tables.sql');
-
-        $sPostcodeFilename = CONST_InstallDir.'/gb_postcode_data.sql.gz';
-        if (file_exists($sPostcodeFilename)) {
-            $this->pgsqlRunScriptFile($sPostcodeFilename);
-        } else {
-            warn('optional external GB postcode table file ('.$sPostcodeFilename.') not found. Skipping.');
-        }
-
-        $sPostcodeFilename = CONST_InstallDir.'/us_postcode_data.sql.gz';
-        if (file_exists($sPostcodeFilename)) {
-            $this->pgsqlRunScriptFile($sPostcodeFilename);
-        } else {
-            warn('optional external US postcode table file ('.$sPostcodeFilename.') not found. Skipping.');
-        }
-
-
-        $this->db()->exec('TRUNCATE location_postcode');
-
-        $sSQL  = 'INSERT INTO location_postcode';
-        $sSQL .= ' (place_id, indexed_status, country_code, postcode, geometry) ';
-        $sSQL .= "SELECT nextval('seq_place'), 1, country_code,";
-        $sSQL .= "       upper(trim (both ' ' from address->'postcode')) as pc,";
-        $sSQL .= '       ST_Centroid(ST_Collect(ST_Centroid(geometry)))';
-        $sSQL .= '  FROM placex';
-        $sSQL .= " WHERE address ? 'postcode' AND address->'postcode' NOT SIMILAR TO '%(,|;)%'";
-        $sSQL .= '       AND geometry IS NOT null';
-        $sSQL .= ' GROUP BY country_code, pc';
-        $this->db()->exec($sSQL);
-
-        // only add postcodes that are not yet available in OSM
-        $sSQL  = 'INSERT INTO location_postcode';
-        $sSQL .= ' (place_id, indexed_status, country_code, postcode, geometry) ';
-        $sSQL .= "SELECT nextval('seq_place'), 1, 'us', postcode,";
-        $sSQL .= '       ST_SetSRID(ST_Point(x,y),4326)';
-        $sSQL .= '  FROM us_postcode WHERE postcode NOT IN';
-        $sSQL .= '        (SELECT postcode FROM location_postcode';
-        $sSQL .= "          WHERE country_code = 'us')";
-        $this->db()->exec($sSQL);
-
-        // add missing postcodes for GB (if available)
-        $sSQL  = 'INSERT INTO location_postcode';
-        $sSQL .= ' (place_id, indexed_status, country_code, postcode, geometry) ';
-        $sSQL .= "SELECT nextval('seq_place'), 1, 'gb', postcode, geometry";
-        $sSQL .= '  FROM gb_postcode WHERE postcode NOT IN';
-        $sSQL .= '           (SELECT postcode FROM location_postcode';
-        $sSQL .= "             WHERE country_code = 'gb')";
-        $this->db()->exec($sSQL);
-
-        if (!$bCMDResultAll) {
-            $sSQL = "DELETE FROM word WHERE class='place' and type='postcode'";
-            $sSQL .= 'and word NOT IN (SELECT postcode FROM location_postcode)';
-            $this->db()->exec($sSQL);
-        }
-
-        $sSQL = 'SELECT count(getorcreate_postcode_id(v)) FROM ';
-        $sSQL .= '(SELECT distinct(postcode) as v FROM location_postcode) p';
-        $this->db()->exec($sSQL);
-    }
-
-    /**
-     * Return the connection to the database.
-     *
-     * @return Database object.
-     *
-     * Creates a new connection if none exists yet. Otherwise reuses the
-     * already established connection.
-     */
-    private function db()
-    {
-        if (is_null($this->oDB)) {
-            $this->oDB = new \Nominatim\DB();
-            $this->oDB->connect();
-        }
-
-        return $this->oDB;
-    }
-
-    private function pgsqlRunScript($sScript, $bfatal = true)
-    {
-        runSQLScript(
-            $sScript,
-            $bfatal,
-            $this->bVerbose,
-            $this->sIgnoreErrors
-        );
-    }
-
-    public function createSqlFunctions()
-    {
-        $oCmd = (clone($this->oNominatimCmd))
-                ->addParams('refresh', '--functions');
-
-        if (!$this->bEnableDiffUpdates) {
-            $oCmd->addParams('--no-diff-updates');
-        }
-
-        if ($this->bEnableDebugStatements) {
-            $oCmd->addParams('--enable-debug-statements');
-        }
-
-        $oCmd->run(!$this->sIgnoreErrors);
-    }
-
-    private function pgsqlRunScriptFile($sFilename)
-    {
-        if (!file_exists($sFilename)) fail('unable to find '.$sFilename);
-
-        $oCmd = (new \Nominatim\Shell('psql'))
-                ->addParams('--port', $this->aDSNInfo['port'])
-                ->addParams('--dbname', $this->aDSNInfo['database']);
-
-        if (!$this->bVerbose) {
-            $oCmd->addParams('--quiet');
-        }
-        if (isset($this->aDSNInfo['hostspec'])) {
-            $oCmd->addParams('--host', $this->aDSNInfo['hostspec']);
-        }
-        if (isset($this->aDSNInfo['username'])) {
-            $oCmd->addParams('--username', $this->aDSNInfo['username']);
-        }
-        if (isset($this->aDSNInfo['password'])) {
-            $oCmd->addEnvPair('PGPASSWORD', $this->aDSNInfo['password']);
-        }
-        $ahGzipPipes = null;
-        if (preg_match('/\\.gz$/', $sFilename)) {
-            $aDescriptors = array(
-                             0 => array('pipe', 'r'),
-                             1 => array('pipe', 'w'),
-                             2 => array('file', '/dev/null', 'a')
-                            );
-            $oZcatCmd = new \Nominatim\Shell('zcat', $sFilename);
-
-            $hGzipProcess = proc_open($oZcatCmd->escapedCmd(), $aDescriptors, $ahGzipPipes);
-            if (!is_resource($hGzipProcess)) fail('unable to start zcat');
-            $aReadPipe = $ahGzipPipes[1];
-            fclose($ahGzipPipes[0]);
-        } else {
-            $oCmd->addParams('--file', $sFilename);
-            $aReadPipe = array('pipe', 'r');
-        }
-        $aDescriptors = array(
-                         0 => $aReadPipe,
-                         1 => array('pipe', 'w'),
-                         2 => array('file', '/dev/null', 'a')
-                        );
-        $ahPipes = null;
-
-        $hProcess = proc_open($oCmd->escapedCmd(), $aDescriptors, $ahPipes, null, $oCmd->aEnv);
-        if (!is_resource($hProcess)) fail('unable to start pgsql');
-        // TODO: error checking
-        while (!feof($ahPipes[1])) {
-            echo fread($ahPipes[1], 4096);
-        }
-        fclose($ahPipes[1]);
-        $iReturn = proc_close($hProcess);
-        if ($iReturn > 0) {
-            fail("pgsql returned with error code ($iReturn)");
-        }
-        if ($ahGzipPipes) {
-            fclose($ahGzipPipes[1]);
-            proc_close($hGzipProcess);
-        }
-    }
-
-    private function replaceSqlPatterns($sSql)
-    {
-        $sSql = str_replace('{www-user}', getSetting('DATABASE_WEBUSER'), $sSql);
-
-        $aPatterns = array(
-                      '{ts:address-data}' => getSetting('TABLESPACE_ADDRESS_DATA'),
-                      '{ts:address-index}' => getSetting('TABLESPACE_ADDRESS_INDEX'),
-                      '{ts:search-data}' => getSetting('TABLESPACE_SEARCH_DATA'),
-                      '{ts:search-index}' =>  getSetting('TABLESPACE_SEARCH_INDEX'),
-                      '{ts:aux-data}' =>  getSetting('TABLESPACE_AUX_DATA'),
-                      '{ts:aux-index}' =>  getSetting('TABLESPACE_AUX_INDEX')
-        );
-
-        foreach ($aPatterns as $sPattern => $sTablespace) {
-            if ($sTablespace) {
-                $sSql = str_replace($sPattern, 'TABLESPACE "'.$sTablespace.'"', $sSql);
-            } else {
-                $sSql = str_replace($sPattern, '', $sSql);
-            }
-        }
-
-        return $sSql;
-    }
-}

lib-php/setup_functions.php

@@ -1,34 +0,0 @@
-<?php
-
-function checkInFile($sOSMFile)
-{
-    if (!isset($sOSMFile)) {
-        fail('missing --osm-file for data import');
-    }
-
-    if (!file_exists($sOSMFile)) {
-        fail('the path supplied to --osm-file does not exist');
-    }
-
-    if (!is_readable($sOSMFile)) {
-        fail('osm-file "' . $aCMDResult['osm-file'] . '" not readable');
-    }
-}
-
-function getOsm2pgsqlBinary()
-{
-    $sBinary = getSetting('OSM2PGSQL_BINARY');
-
-    return $sBinary ? $sBinary : CONST_Default_Osm2pgsql;
-}
-
-function getImportStyle()
-{
-    $sStyle = getSetting('IMPORT_STYLE');
-
-    if (in_array($sStyle, array('admin', 'street', 'address', 'full', 'extratags'))) {
-        return CONST_ConfigDir.'/import-'.$sStyle.'.style';
-    }
-
-    return $sStyle;
-}

lib-sql/functions.sql

@@ -1,20 +0,0 @@
-{% include('functions/utils.sql') %}
-{% include('functions/normalization.sql') %}
-{% include('functions/ranking.sql') %}
-{% include('functions/importance.sql') %}
-{% include('functions/address_lookup.sql') %}
-{% include('functions/interpolation.sql') %}
-
-{% if 'place' in db.tables %}
-    {% include 'functions/place_triggers.sql' %}
-{% endif %}
-
-{% if 'placex' in db.tables %}
-    {% include 'functions/placex_triggers.sql' %}
-{% endif %}
-
-{% if 'location_postcode' in db.tables %}
-    {% include 'functions/postcode_triggers.sql' %}
-{% endif %}
-
-{% include('functions/partition-functions.sql') %}

lib-sql/indices.sql

@@ -1,71 +0,0 @@
--- Indices used only during search and update.
--- These indices are created only after the indexing process is done.
-
-CREATE INDEX {{sql.if_index_not_exists}} idx_word_word_id
-  ON word USING BTREE (word_id) {{db.tablespace.search_index}};
-
-CREATE INDEX {{sql.if_index_not_exists}} idx_place_addressline_address_place_id
-  ON place_addressline USING BTREE (address_place_id) {{db.tablespace.search_index}};
-
-CREATE INDEX {{sql.if_index_not_exists}} idx_placex_rank_search
-  ON placex USING BTREE (rank_search) {{db.tablespace.search_index}};
-
-CREATE INDEX {{sql.if_index_not_exists}} idx_placex_rank_address
-  ON placex USING BTREE (rank_address) {{db.tablespace.search_index}};
-
-CREATE INDEX {{sql.if_index_not_exists}} idx_placex_parent_place_id
-  ON placex USING BTREE (parent_place_id) {{db.tablespace.search_index}}
-  WHERE parent_place_id IS NOT NULL;
-
-CREATE INDEX {{sql.if_index_not_exists}} idx_placex_geometry_reverse_lookupPolygon
-  ON placex USING gist (geometry) {{db.tablespace.search_index}}
-  WHERE St_GeometryType(geometry) in ('ST_Polygon', 'ST_MultiPolygon')
-    AND rank_address between 4 and 25 AND type != 'postcode'
-    AND name is not null AND indexed_status = 0 AND linked_place_id is null;
-
-CREATE INDEX {{sql.if_index_not_exists}} idx_placex_geometry_reverse_placeNode
-  ON placex USING gist (geometry) {{db.tablespace.search_index}}
-  WHERE osm_type = 'N' AND rank_search between 5 and 25
-    AND class = 'place' AND type != 'postcode'
-    AND name is not null AND indexed_status = 0 AND linked_place_id is null;
-
-CREATE INDEX {{sql.if_index_not_exists}} idx_osmline_parent_place_id
-  ON location_property_osmline USING BTREE (parent_place_id) {{db.tablespace.search_index}};
-
-CREATE INDEX {{sql.if_index_not_exists}} idx_osmline_parent_osm_id
-  ON location_property_osmline USING BTREE (osm_id) {{db.tablespace.search_index}};
-
-CREATE INDEX {{sql.if_index_not_exists}} idx_postcode_postcode
-  ON location_postcode USING BTREE (postcode) {{db.tablespace.search_index}};
-
--- Indices only needed for updating.
-
-{% if not drop %}
-  CREATE INDEX {{sql.if_index_not_exists}} idx_placex_pendingsector
-    ON placex USING BTREE (rank_address,geometry_sector) {{db.tablespace.address_index}}
-    WHERE indexed_status > 0;
-
-  CREATE INDEX {{sql.if_index_not_exists}} idx_location_area_country_place_id
-    ON location_area_country USING BTREE (place_id) {{db.tablespace.address_index}};
-
-  CREATE UNIQUE INDEX {{sql.if_index_not_exists}} idx_place_osm_unique
-    ON place USING btree(osm_id, osm_type, class, type) {{db.tablespace.address_index}};
-{% endif %}
-
--- Indices only needed for search.
-
-{% if 'search_name' in db.tables %}
-  CREATE INDEX {{sql.if_index_not_exists}} idx_search_name_nameaddress_vector
-    ON search_name USING GIN (nameaddress_vector) WITH (fastupdate = off) {{db.tablespace.search_index}};
-  CREATE INDEX {{sql.if_index_not_exists}} idx_search_name_name_vector
-    ON search_name USING GIN (name_vector) WITH (fastupdate = off) {{db.tablespace.search_index}};
-  CREATE INDEX {{sql.if_index_not_exists}} idx_search_name_centroid
-    ON search_name USING GIST (centroid) {{db.tablespace.search_index}};
-
-  {% if postgres.has_index_non_key_column %}
-    CREATE INDEX {{sql.if_index_not_exists}} idx_placex_housenumber
-      ON placex USING btree (parent_place_id) INCLUDE (housenumber) WHERE housenumber is not null;
-    CREATE INDEX {{sql.if_index_not_exists}} idx_osmline_parent_osm_id_with_hnr
-      ON location_property_osmline USING btree(parent_place_id) INCLUDE (startnumber, endnumber);
-  {% endif %}
-{% endif %}

lib-sql/partition-tables.src.sql

@@ -1,30 +0,0 @@
-drop table IF EXISTS search_name_blank CASCADE;
-CREATE TABLE search_name_blank (
-  place_id BIGINT,
-  address_rank smallint,
-  name_vector integer[],
-  centroid GEOMETRY(Geometry, 4326)
-  );
-
-
-{% for partition in db.partitions %}
-  CREATE TABLE location_area_large_{{ partition }} () INHERITS (location_area_large) {{db.tablespace.address_data}};
-  CREATE INDEX idx_location_area_large_{{ partition }}_place_id ON location_area_large_{{ partition }} USING BTREE (place_id) {{db.tablespace.address_index}};
-  CREATE INDEX idx_location_area_large_{{ partition }}_geometry ON location_area_large_{{ partition }} USING GIST (geometry) {{db.tablespace.address_index}};
-
-  CREATE TABLE search_name_{{ partition }} () INHERITS (search_name_blank) {{db.tablespace.address_data}};
-  CREATE INDEX idx_search_name_{{ partition }}_place_id ON search_name_{{ partition }} USING BTREE (place_id) {{db.tablespace.address_index}};
-  CREATE INDEX idx_search_name_{{ partition }}_centroid_street ON search_name_{{ partition }} USING GIST (centroid) {{db.tablespace.address_index}} where address_rank between 26 and 27;
-  CREATE INDEX idx_search_name_{{ partition }}_centroid_place ON search_name_{{ partition }} USING GIST (centroid) {{db.tablespace.address_index}} where address_rank between 2 and 25;
-
-  DROP TABLE IF EXISTS location_road_{{ partition }};
-  CREATE TABLE location_road_{{ partition }} (
-    place_id BIGINT,
-    partition SMALLINT,
-    country_code VARCHAR(2),
-    geometry GEOMETRY(Geometry, 4326)
-    ) {{db.tablespace.address_data}};
-  CREATE INDEX idx_location_road_{{ partition }}_geometry ON location_road_{{ partition }} USING GIST (geometry) {{db.tablespace.address_index}};
-  CREATE INDEX idx_location_road_{{ partition }}_place_id ON location_road_{{ partition }} USING BTREE (place_id) {{db.tablespace.address_index}};
-
-{% endfor %}

lib-sql/postcode_tables.sql

@@ -1,15 +0,0 @@
-DROP TABLE IF EXISTS gb_postcode;
-CREATE TABLE gb_postcode (
-    id integer,
-    postcode character varying(9),
-    geometry geometry,
-    CONSTRAINT enforce_dims_geometry CHECK ((st_ndims(geometry) = 2)),
-    CONSTRAINT enforce_srid_geometry CHECK ((st_srid(geometry) = 4326))
-);
-
-DROP TABLE IF EXISTS us_postcode;
-CREATE TABLE us_postcode (
-    postcode text,
-    x double precision,
-    y double precision
-);

lib-sql/tiger_import_finish.sql

@@ -1,15 +0,0 @@
---index only on parent_place_id
-CREATE INDEX {{sql.if_index_not_exists}} idx_location_property_tiger_place_id_imp
-  ON location_property_tiger_import (parent_place_id) {{db.tablespace.aux_index}};
-CREATE UNIQUE INDEX {{sql.if_index_not_exists}} idx_location_property_tiger_place_id_imp
-  ON location_property_tiger_import (place_id) {{db.tablespace.aux_index}};
-
-GRANT SELECT ON location_property_tiger_import TO "{{config.DATABASE_WEBUSER}}";
-
-DROP TABLE IF EXISTS location_property_tiger;
-ALTER TABLE location_property_tiger_import RENAME TO location_property_tiger;
-
-ALTER INDEX IF EXISTS idx_location_property_tiger_parent_place_id_imp RENAME TO idx_location_property_tiger_housenumber_parent_place_id;
-ALTER INDEX IF EXISTS idx_location_property_tiger_place_id_imp RENAME TO idx_location_property_tiger_place_id;
-
-DROP FUNCTION tiger_line_import (linegeo geometry, in_startnumber integer, in_endnumber integer, interpolationtype text, in_street text, in_isin text, in_postcode text);

lib/AddressDetails.php

@@ -2,7 +2,7 @@
 
 namespace Nominatim;
 
-require_once(CONST_LibDir.'/ClassTypes.php');
+require_once(CONST_BasePath.'/lib/ClassTypes.php');
 
 /**
  * Detailed list of address parts for a single result

lib/DB.php

@@ -2,7 +2,7 @@
 
 namespace Nominatim;
 
-require_once(CONST_LibDir.'/DatabaseError.php');
+require_once(CONST_BasePath.'/lib/DatabaseError.php');
 
 /**
  * Uses PDO to access the database specified in the CONST_Database_DSN
@@ -12,9 +12,9 @@ class DB
 {
     protected $connection;
 
-    public function __construct($sDSN = null)
+    public function __construct($sDSN = CONST_Database_DSN)
     {
-        $this->sDSN = $sDSN ?? getSetting('DATABASE_DSN');
+        $this->sDSN = $sDSN;
     }
 
     public function connect($bNew = false, $bPersistent = true)
@@ -240,6 +240,16 @@ class DB
         return ($this->getOne($sSQL, array(':tablename' => $sTableName)) == 1);
     }
 
+    /**
+    * Returns a list of table names in the database
+    *
+    * @return array[]
+    */
+    public function getListOfTables()
+    {
+        return $this->getCol("SELECT tablename FROM pg_tables WHERE schemaname='public'");
+    }
+
     /**
      * Deletes a table. Returns true if deleted or didn't exist.
      *
@@ -252,6 +262,76 @@ class DB
         return $this->exec('DROP TABLE IF EXISTS '.$sTableName.' CASCADE') == 0;
     }
 
+    /**
+    * Check if an index exists in the database. Optional filtered by tablename
+    *
+    * @param string  $sTableName
+    *
+    * @return boolean
+    */
+    public function indexExists($sIndexName, $sTableName = null)
+    {
+        return in_array($sIndexName, $this->getListOfIndices($sTableName));
+    }
+
+    /**
+    * Returns a list of index names in the database, optional filtered by tablename
+    *
+    * @param string  $sTableName
+    *
+    * @return array
+    */
+    public function getListOfIndices($sTableName = null)
+    {
+        //  table_name            | index_name                      | column_name
+        // -----------------------+---------------------------------+--------------
+        //  country_name          | idx_country_name_country_code   | country_code
+        //  country_osm_grid      | idx_country_osm_grid_geometry   | geometry
+        //  import_polygon_delete | idx_import_polygon_delete_osmid | osm_id
+        //  import_polygon_delete | idx_import_polygon_delete_osmid | osm_type
+        //  import_polygon_error  | idx_import_polygon_error_osmid  | osm_id
+        //  import_polygon_error  | idx_import_polygon_error_osmid  | osm_type
+        $sSql = <<< END
+SELECT
+    t.relname as table_name,
+    i.relname as index_name,
+    a.attname as column_name
+FROM
+    pg_class t,
+    pg_class i,
+    pg_index ix,
+    pg_attribute a
+WHERE
+    t.oid = ix.indrelid
+    and i.oid = ix.indexrelid
+    and a.attrelid = t.oid
+    and a.attnum = ANY(ix.indkey)
+    and t.relkind = 'r'
+    and i.relname NOT LIKE 'pg_%'
+    FILTERS
+ ORDER BY
+    t.relname,
+    i.relname,
+    a.attname
+END;
+
+        $aRows = null;
+        if ($sTableName) {
+            $sSql = str_replace('FILTERS', 'and t.relname = :tablename', $sSql);
+            $aRows = $this->getAll($sSql, array(':tablename' => $sTableName));
+        } else {
+            $sSql = str_replace('FILTERS', '', $sSql);
+            $aRows = $this->getAll($sSql);
+        }
+
+        $aIndexNames = array_unique(array_map(function ($aRow) {
+            return $aRow['index_name'];
+        }, $aRows));
+        sort($aIndexNames);
+
+        return $aIndexNames;
+    }
+
     /**
      * Tries to connect to the database but on failure doesn't throw an exception.
      *

lib/Geocode.php

@@ -2,12 +2,12 @@
 
 namespace Nominatim;
 
-require_once(CONST_LibDir.'/PlaceLookup.php');
-require_once(CONST_LibDir.'/Phrase.php');
-require_once(CONST_LibDir.'/ReverseGeocode.php');
-require_once(CONST_LibDir.'/SearchDescription.php');
-require_once(CONST_LibDir.'/SearchContext.php');
-require_once(CONST_LibDir.'/TokenList.php');
+require_once(CONST_BasePath.'/lib/PlaceLookup.php');
+require_once(CONST_BasePath.'/lib/Phrase.php');
+require_once(CONST_BasePath.'/lib/ReverseGeocode.php');
+require_once(CONST_BasePath.'/lib/SearchDescription.php');
+require_once(CONST_BasePath.'/lib/SearchContext.php');
+require_once(CONST_BasePath.'/lib/TokenList.php');
 
 class Geocode
 {
@@ -18,7 +18,7 @@ class Geocode
     protected $aLangPrefOrder = array();
 
     protected $aExcludePlaceIDs = array();
-    protected $bReverseInPlan = true;
+    protected $bReverseInPlan = false;
 
     protected $iLimit = 20;
     protected $iFinalLimit = 10;
@@ -778,19 +778,14 @@ class Geocode
                 if (!empty($aResults)) {
                     $aSplitResults = Result::splitResults($aResults);
                     Debug::printVar('Split results', $aSplitResults);
-                    if ($iGroupLoop <= 4
-                        && reset($aSplitResults['head'])->iResultRank > 0
-                        && $iGroupedRank !== array_key_last($aGroupedSearches)) {
+                    if ($iGroupLoop <= 4 && empty($aSplitResults['tail'])
+                        && reset($aSplitResults['head'])->iResultRank > 0) {
                         // Haven't found an exact match for the query yet.
                         // Therefore add result from the next group level.
                         $aNextResults = $aSplitResults['head'];
                         foreach ($aNextResults as $oRes) {
                             $oRes->iResultRank--;
                         }
-                        foreach ($aSplitResults['tail'] as $oRes) {
-                            $oRes->iResultRank--;
-                            $aNextResults[$oRes->iId] = $oRes;
-                        }
                         $aResults = array();
                     } else {
                         $aResults = $aSplitResults['head'];

lib/PlaceLookup.php

@@ -2,8 +2,8 @@
 
 namespace Nominatim;
 
-require_once(CONST_LibDir.'/AddressDetails.php');
-require_once(CONST_LibDir.'/Result.php');
+require_once(CONST_BasePath.'/lib/AddressDetails.php');
+require_once(CONST_BasePath.'/lib/Result.php');
 
 class PlaceLookup
 {
@@ -486,63 +486,65 @@ class PlaceLookup
         $aOutlineResult = array();
         if (!$iPlaceID) return $aOutlineResult;
 
-        // Get the bounding box and outline polygon
-        $sSQL = 'select place_id,0 as numfeatures,st_area(geometry) as area,';
-        if ($fLonReverse != null && $fLatReverse != null) {
-            $sSQL .= ' ST_Y(closest_point) as centrelat,';
-            $sSQL .= ' ST_X(closest_point) as centrelon,';
-        } else {
-            $sSQL .= ' ST_Y(centroid) as centrelat, ST_X(centroid) as centrelon,';
-        }
-        $sSQL .= ' ST_YMin(geometry) as minlat,ST_YMax(geometry) as maxlat,';
-        $sSQL .= ' ST_XMin(geometry) as minlon,ST_XMax(geometry) as maxlon';
-        if ($this->bIncludePolygonAsGeoJSON) $sSQL .= ',ST_AsGeoJSON(geometry) as asgeojson';
-        if ($this->bIncludePolygonAsKML) $sSQL .= ',ST_AsKML(geometry) as askml';
-        if ($this->bIncludePolygonAsSVG) $sSQL .= ',ST_AsSVG(geometry) as assvg';
-        if ($this->bIncludePolygonAsText) $sSQL .= ',ST_AsText(geometry) as astext';
-        if ($fLonReverse != null && $fLatReverse != null) {
-            $sFrom = ' from (SELECT * , CASE WHEN (class = \'highway\') AND (ST_GeometryType(geometry) = \'ST_LineString\') THEN ';
-            $sFrom .=' ST_ClosestPoint(geometry, ST_SetSRID(ST_Point('.$fLatReverse.','.$fLonReverse.'),4326))';
-            $sFrom .=' ELSE centroid END AS closest_point';
-            $sFrom .= ' from placex where place_id = '.$iPlaceID.') as plx';
-        } else {
-            $sFrom = ' from placex where place_id = '.$iPlaceID;
-        }
-        if ($this->fPolygonSimplificationThreshold > 0) {
-            $sSQL .= ' from (select place_id,centroid,ST_SimplifyPreserveTopology(geometry,'.$this->fPolygonSimplificationThreshold.') as geometry'.$sFrom.') as plx';
-        } else {
-            $sSQL .= $sFrom;
-        }
-
-        $aPointPolygon = $this->oDB->getRow($sSQL, null, 'Could not get outline');
-
-        if ($aPointPolygon && $aPointPolygon['place_id']) {
-            if ($aPointPolygon['centrelon'] !== null && $aPointPolygon['centrelat'] !== null) {
-                $aOutlineResult['lat'] = $aPointPolygon['centrelat'];
-                $aOutlineResult['lon'] = $aPointPolygon['centrelon'];
+        if (CONST_Search_AreaPolygons) {
+            // Get the bounding box and outline polygon
+            $sSQL = 'select place_id,0 as numfeatures,st_area(geometry) as area,';
+            if ($fLonReverse != null && $fLatReverse != null) {
+                $sSQL .= ' ST_Y(closest_point) as centrelat,';
+                $sSQL .= ' ST_X(closest_point) as centrelon,';
+            } else {
+                $sSQL .= ' ST_Y(centroid) as centrelat, ST_X(centroid) as centrelon,';
+            }
+            $sSQL .= ' ST_YMin(geometry) as minlat,ST_YMax(geometry) as maxlat,';
+            $sSQL .= ' ST_XMin(geometry) as minlon,ST_XMax(geometry) as maxlon';
+            if ($this->bIncludePolygonAsGeoJSON) $sSQL .= ',ST_AsGeoJSON(geometry) as asgeojson';
+            if ($this->bIncludePolygonAsKML) $sSQL .= ',ST_AsKML(geometry) as askml';
+            if ($this->bIncludePolygonAsSVG) $sSQL .= ',ST_AsSVG(geometry) as assvg';
+            if ($this->bIncludePolygonAsText) $sSQL .= ',ST_AsText(geometry) as astext';
+            if ($fLonReverse != null && $fLatReverse != null) {
+                $sFrom = ' from (SELECT * , CASE WHEN (class = \'highway\') AND (ST_GeometryType(geometry) = \'ST_LineString\') THEN ';
+                $sFrom .=' ST_ClosestPoint(geometry, ST_SetSRID(ST_Point('.$fLatReverse.','.$fLonReverse.'),4326))';
+                $sFrom .=' ELSE centroid END AS closest_point';
+                $sFrom .= ' from placex where place_id = '.$iPlaceID.') as plx';
+            } else {
+                $sFrom = ' from placex where place_id = '.$iPlaceID;
+            }
+            if ($this->fPolygonSimplificationThreshold > 0) {
+                $sSQL .= ' from (select place_id,centroid,ST_SimplifyPreserveTopology(geometry,'.$this->fPolygonSimplificationThreshold.') as geometry'.$sFrom.') as plx';
+            } else {
+                $sSQL .= $sFrom;
             }
 
-            if ($this->bIncludePolygonAsGeoJSON) $aOutlineResult['asgeojson'] = $aPointPolygon['asgeojson'];
-            if ($this->bIncludePolygonAsKML) $aOutlineResult['askml'] = $aPointPolygon['askml'];
-            if ($this->bIncludePolygonAsSVG) $aOutlineResult['assvg'] = $aPointPolygon['assvg'];
-            if ($this->bIncludePolygonAsText) $aOutlineResult['astext'] = $aPointPolygon['astext'];
+            $aPointPolygon = $this->oDB->getRow($sSQL, null, 'Could not get outline');
 
-            if (abs($aPointPolygon['minlat'] - $aPointPolygon['maxlat']) < 0.0000001) {
-                $aPointPolygon['minlat'] = $aPointPolygon['minlat'] - $fRadius;
-                $aPointPolygon['maxlat'] = $aPointPolygon['maxlat'] + $fRadius;
+            if ($aPointPolygon && $aPointPolygon['place_id']) {
+                if ($aPointPolygon['centrelon'] !== null && $aPointPolygon['centrelat'] !== null) {
+                    $aOutlineResult['lat'] = $aPointPolygon['centrelat'];
+                    $aOutlineResult['lon'] = $aPointPolygon['centrelon'];
+                }
+
+                if ($this->bIncludePolygonAsGeoJSON) $aOutlineResult['asgeojson'] = $aPointPolygon['asgeojson'];
+                if ($this->bIncludePolygonAsKML) $aOutlineResult['askml'] = $aPointPolygon['askml'];
+                if ($this->bIncludePolygonAsSVG) $aOutlineResult['assvg'] = $aPointPolygon['assvg'];
+                if ($this->bIncludePolygonAsText) $aOutlineResult['astext'] = $aPointPolygon['astext'];
+
+                if (abs($aPointPolygon['minlat'] - $aPointPolygon['maxlat']) < 0.0000001) {
+                    $aPointPolygon['minlat'] = $aPointPolygon['minlat'] - $fRadius;
+                    $aPointPolygon['maxlat'] = $aPointPolygon['maxlat'] + $fRadius;
+                }
+
+                if (abs($aPointPolygon['minlon'] - $aPointPolygon['maxlon']) < 0.0000001) {
+                    $aPointPolygon['minlon'] = $aPointPolygon['minlon'] - $fRadius;
+                    $aPointPolygon['maxlon'] = $aPointPolygon['maxlon'] + $fRadius;
+                }
+
+                $aOutlineResult['aBoundingBox'] = array(
+                                                   (string)$aPointPolygon['minlat'],
+                                                   (string)$aPointPolygon['maxlat'],
+                                                   (string)$aPointPolygon['minlon'],
+                                                   (string)$aPointPolygon['maxlon']
+                                                  );
             }
-
-            if (abs($aPointPolygon['minlon'] - $aPointPolygon['maxlon']) < 0.0000001) {
-                $aPointPolygon['minlon'] = $aPointPolygon['minlon'] - $fRadius;
-                $aPointPolygon['maxlon'] = $aPointPolygon['maxlon'] + $fRadius;
-            }
-
-            $aOutlineResult['aBoundingBox'] = array(
-                                               (string)$aPointPolygon['minlat'],
-                                               (string)$aPointPolygon['maxlat'],
-                                               (string)$aPointPolygon['minlon'],
-                                               (string)$aPointPolygon['maxlon']
-                                              );
         }
 
         // as a fallback we generate a bounding box without knowing the size of the geometry

lib/Result.php

@@ -26,8 +26,6 @@ class Result
     public $iExactMatches = 0;
     /// Subranking within the results (the higher the worse).
     public $iResultRank = 0;
-    /// Address rank of the result.
-    public $iAddressRank;
 
     public function debugInfo()
     {
@@ -86,7 +84,7 @@ class Result
 
         foreach ($aResults as $oRes) {
             if ($oRes->iResultRank < $iMinRank) {
-                $aTail += $aHead;
+                $aTail = array_merge($aTail, $aHead);
                 $aHead = array($oRes->iId => $oRes);
                 $iMinRank = $oRes->iResultRank;
             } elseif ($oRes->iResultRank == $iMinRank) {

lib/ReverseGeocode.php

@@ -2,7 +2,7 @@
 
 namespace Nominatim;
 
-require_once(CONST_LibDir.'/Result.php');
+require_once(CONST_BasePath.'/lib/Result.php');
 
 class ReverseGeocode
 {
@@ -280,6 +280,29 @@ class ReverseGeocode
                 $iPlaceID = $aPlace['place_id'];
                 $oResult = new Result($iPlaceID);
                 $iRankAddress = $aPlace['rank_address'];
+                $iParentPlaceID = $aPlace['parent_place_id'];
+            }
+
+            if ($bDoInterpolation && $iMaxRank >= 30) {
+                $fDistance = $fSearchDiam;
+                if ($aPlace) {
+                    // We can't reliably go from the closest street to an
+                    // interpolation line because the closest interpolation
+                    // may have a different street segments as a parent.
+                    // Therefore allow an interpolation line to take precendence
+                    // even when the street is closer.
+                    $fDistance = $iRankAddress < 28 ? 0.001 : $aPlace['distance'];
+                }
+
+                $aHouse = $this->lookupInterpolation($sPointSQL, $fDistance);
+                Debug::printVar('Interpolation result', $aPlace);
+
+                if ($aHouse) {
+                    $oResult = new Result($aHouse['place_id'], Result::TABLE_OSMLINE);
+                    $oResult->iHouseNumber = closestHouseNumber($aHouse);
+                    $aPlace = $aHouse;
+                    $iRankAddress = 30;
+                }
             }
 
             if ($aPlace) {
@@ -305,9 +328,7 @@ class ReverseGeocode
                     Debug::printVar('Closest POI result', $aStreet);
 
                     if ($aStreet) {
-                        $aPlace = $aStreet;
                         $oResult = new Result($aStreet['place_id']);
-                        $iRankAddress = 30;
                     }
                 }
 
@@ -330,37 +351,11 @@ class ReverseGeocode
                     Debug::printVar('Tiger house number result', $aPlaceTiger);
 
                     if ($aPlaceTiger) {
-                        $aPlace = $aPlaceTiger;
                         $oResult = new Result($aPlaceTiger['place_id'], Result::TABLE_TIGER);
                         $oResult->iHouseNumber = closestHouseNumber($aPlaceTiger);
-                        $iRankAddress = 30;
                     }
                 }
-            }
-
-            if ($bDoInterpolation && $iMaxRank >= 30) {
-                $fDistance = $fSearchDiam;
-                if ($aPlace) {
-                    // We can't reliably go from the closest street to an
-                    // interpolation line because the closest interpolation
-                    // may have a different street segments as a parent.
-                    // Therefore allow an interpolation line to take precendence
-                    // even when the street is closer.
-                    $fDistance = $iRankAddress < 28 ? 0.001 : $aPlace['distance'];
-                }
-
-                $aHouse = $this->lookupInterpolation($sPointSQL, $fDistance);
-                Debug::printVar('Interpolation result', $aPlace);
-
-                if ($aHouse) {
-                    $oResult = new Result($aHouse['place_id'], Result::TABLE_OSMLINE);
-                    $oResult->iHouseNumber = closestHouseNumber($aHouse);
-                    $aPlace = $aHouse;
-                    $iRankAddress = 30;
-                }
-            }
-
-            if (!$aPlace) {
+            } else {
                 // if no POI or street is found ...
                 $oResult = $this->lookupLargeArea($sPointSQL, 25);
             }

lib/SearchContext.php

@@ -2,7 +2,7 @@
 
 namespace Nominatim;
 
-require_once(CONST_LibDir.'/lib.php');
+require_once(CONST_BasePath.'/lib/lib.php');
 
 
 /**

lib/SearchDescription.php

@@ -2,9 +2,9 @@
 
 namespace Nominatim;
 
-require_once(CONST_LibDir.'/SpecialSearchOperator.php');
-require_once(CONST_LibDir.'/SearchContext.php');
-require_once(CONST_LibDir.'/Result.php');
+require_once(CONST_BasePath.'/lib/SpecialSearchOperator.php');
+require_once(CONST_BasePath.'/lib/SearchContext.php');
+require_once(CONST_BasePath.'/lib/Result.php');
 
 /**
  * Description of a single interpretation of a search query.
@@ -86,6 +86,18 @@ class SearchDescription
         $this->sType = $sType;
     }
 
+    /**
+     * Check if this might be a full address search.
+     *
+     * @return bool True if the search contains name, address and housenumber.
+     */
+    public function looksLikeFullAddress()
+    {
+        return (!empty($this->aName))
+               && (!empty($this->aAddress) || $this->sCountryCode)
+               && preg_match('/[0-9]+/', $this->sHouseNumber);
+    }
+
     /**
      * Check if any operator is set.
      *
@@ -179,7 +191,6 @@ class SearchDescription
                 // - increase score for finding it anywhere else (optimisation)
                 if (!$bLastToken) {
                     $oSearch->iSearchRank += 5;
-                    $oSearch->iNamePhrase = -1;
                 }
                 $aNewSearches[] = $oSearch;
             }
@@ -206,7 +217,6 @@ class SearchDescription
                 ) {
                     $oSearch = clone $this;
                     $oSearch->iSearchRank++;
-                    $oSearch->iNamePhrase = -1;
                     if (strlen($oSearchTerm->sPostcode) < 4) {
                         $oSearch->iSearchRank += 4 - strlen($oSearchTerm->sPostcode);
                     }
@@ -220,11 +230,7 @@ class SearchDescription
             if (!$this->sHouseNumber && $this->iOperator != Operator::POSTCODE) {
                 $oSearch = clone $this;
                 $oSearch->iSearchRank++;
-                $oSearch->iNamePhrase = -1;
                 $oSearch->sHouseNumber = $oSearchTerm->sToken;
-                if ($this->iOperator != Operator::NONE) {
-                    $oSearch->iSearchRank++;
-                }
                 // sanity check: if the housenumber is not mainly made
                 // up of numbers, add a penalty
                 if (preg_match('/\\d/', $oSearch->sHouseNumber) === 0
@@ -261,8 +267,7 @@ class SearchDescription
         ) {
             if ($this->iOperator == Operator::NONE) {
                 $oSearch = clone $this;
-                $oSearch->iSearchRank += 2;
-                $oSearch->iNamePhrase = -1;
+                $oSearch->iSearchRank++;
 
                 $iOp = $oSearchTerm->iOperator;
                 if ($iOp == Operator::NONE) {
@@ -272,11 +277,6 @@ class SearchDescription
                         $iOp = Operator::NEAR;
                     }
                     $oSearch->iSearchRank += 2;
-                } elseif (!$bFirstToken && !$bLastToken) {
-                    $oSearch->iSearchRank += 2;
-                }
-                if ($this->sHouseNumber) {
-                    $oSearch->iSearchRank++;
                 }
 
                 $oSearch->setPoiSearch(
@@ -297,12 +297,11 @@ class SearchDescription
             if (!empty($this->aName) || !($bFirstPhrase || $sPhraseType == '')) {
                 if (($sPhraseType == '' || !$bFirstPhrase) && !$bHasPartial) {
                     $oSearch = clone $this;
-                    $oSearch->iNamePhrase = -1;
                     $oSearch->iSearchRank += 3 * $oSearchTerm->iTermCount;
                     $oSearch->aAddress[$iWordID] = $iWordID;
                     $aNewSearches[] = $oSearch;
                 }
-            } elseif (empty($this->aNameNonSearch)) {
+            } else {
                 $oSearch = clone $this;
                 $oSearch->iSearchRank++;
                 $oSearch->aName = array($iWordID => $iWordID);
@@ -342,34 +341,51 @@ class SearchDescription
 
         if ((!$bStructuredPhrases || $iPhrase > 0)
             && (!empty($this->aName))
+            && strpos($sToken, ' ') === false
         ) {
-            $oSearch = clone $this;
-            $oSearch->iSearchRank++;
-            if (preg_match('#^[0-9 ]+$#', $sToken)) {
-                $oSearch->iSearchRank++;
-            }
             if ($oSearchTerm->iSearchNameCount < CONST_Max_Word_Frequency) {
+                $oSearch = clone $this;
+                $oSearch->iSearchRank += $oSearchTerm->iTermCount + 1;
+                if (empty($this->aName)) {
+                    $oSearch->iSearchRank++;
+                }
+                if (preg_match('#^[0-9]+$#', $sToken)) {
+                    $oSearch->iSearchRank++;
+                }
                 $oSearch->aAddress[$iWordID] = $iWordID;
+                $aNewSearches[] = $oSearch;
             } else {
+                $oSearch = clone $this;
+                $oSearch->iSearchRank += $oSearchTerm->iTermCount + 1;
                 $oSearch->aAddressNonSearch[$iWordID] = $iWordID;
                 if (!empty($aFullTokens)) {
                     $oSearch->iSearchRank++;
                 }
+                $aNewSearches[] = $oSearch;
+
+                // revert to the token version?
+                foreach ($aFullTokens as $oSearchTermToken) {
+                    if (is_a($oSearchTermToken, '\Nominatim\Token\Word')) {
+                        $oSearch = clone $this;
+                        $oSearch->iSearchRank += 3;
+                        $oSearch->aAddress[$oSearchTermToken->iId]
+                            = $oSearchTermToken->iId;
+                        $aNewSearches[] = $oSearch;
+                    }
+                }
             }
-            $aNewSearches[] = $oSearch;
         }
 
         if ((!$this->sPostcode && !$this->aAddress && !$this->aAddressNonSearch)
-            && ((empty($this->aName) && empty($this->aNameNonSearch)) || $this->iNamePhrase == $iPhrase)
-            && strpos($sToken, ' ') === false
+            && (empty($this->aName) || $this->iNamePhrase == $iPhrase)
         ) {
             $oSearch = clone $this;
-            $oSearch->iSearchRank++;
-            if (empty($this->aName) && empty($this->aNameNonSearch)) {
-                $oSearch->iSearchRank++;
+            $oSearch->iSearchRank += 2;
+            if (empty($this->aName)) {
+                $oSearch->iSearchRank += 1;
             }
-            if (preg_match('#^[0-9 ]+$#', $sToken)) {
-                $oSearch->iSearchRank++;
+            if (preg_match('#^[0-9]+$#', $sToken)) {
+                $oSearch->iSearchRank += 2;
             }
             if ($oSearchTerm->iSearchNameCount < CONST_Max_Word_Frequency) {
                 if (empty($this->aName)
@@ -383,9 +399,6 @@ class SearchDescription
                 }
                 $oSearch->aName[$iWordID] = $iWordID;
             } else {
-                if (!empty($aFullTokens)) {
-                    $oSearch->iSearchRank++;
-                }
                 $oSearch->aNameNonSearch[$iWordID] = $iWordID;
             }
             $oSearch->iNamePhrase = $iPhrase;
@@ -449,11 +462,7 @@ class SearchDescription
                 // Downgrade the rank of the street results, they are missing
                 // the housenumber.
                 foreach ($aResults as $oRes) {
-                    if ($oRes->iAddressRank >= 26) {
-                        $oRes->iResultRank++;
-                    } else {
-                        $oRes->iResultRank += 2;
-                    }
+                    $oRes->iResultRank++;
                 }
 
                 $aHnResults = $this->queryHouseNumber($oDB, $aResults);
@@ -614,14 +623,14 @@ class SearchDescription
         // too many results are expected for the street, i.e. if the result
         // will be narrowed down by an address. Remeber that with ordering
         // every single result has to be checked.
-        if ($this->sHouseNumber && ($this->bRareName || !empty($this->aAddress) || $this->sPostcode)) {
+        if ($this->sHouseNumber && (!empty($this->aAddress) || $this->sPostcode)) {
             $sHouseNumberRegex = '\\\\m'.$this->sHouseNumber.'\\\\M';
             $aOrder[] = ' (';
             $aOrder[0] .= 'EXISTS(';
             $aOrder[0] .= '  SELECT place_id';
             $aOrder[0] .= '  FROM placex';
             $aOrder[0] .= '  WHERE parent_place_id = search_name.place_id';
-            $aOrder[0] .= "    AND housenumber ~* E'".$sHouseNumberRegex."'";
+            $aOrder[0] .= "    AND transliteration(housenumber) ~* E'".$sHouseNumberRegex."'";
             $aOrder[0] .= '  LIMIT 1';
             $aOrder[0] .= ') ';
             // also housenumbers from interpolation lines table are needed
@@ -718,7 +727,7 @@ class SearchDescription
         $aResults = array();
 
         if (!empty($aTerms)) {
-            $sSQL = 'SELECT place_id, address_rank,'.$sExactMatchSQL;
+            $sSQL = 'SELECT place_id,'.$sExactMatchSQL;
             $sSQL .= ' FROM search_name';
             $sSQL .= ' WHERE '.join(' and ', $aTerms);
             $sSQL .= ' ORDER BY '.join(', ', $aOrder);
@@ -731,7 +740,6 @@ class SearchDescription
             foreach ($aDBResults as $aResult) {
                 $oResult = new Result($aResult['place_id']);
                 $oResult->iExactMatches = $aResult['exactmatch'];
-                $oResult->iAddressRank = $aResult['address_rank'];
                 $aResults[$aResult['place_id']] = $oResult;
             }
         }
@@ -751,7 +759,7 @@ class SearchDescription
         $sHouseNumberRegex = '\\\\m'.$this->sHouseNumber.'\\\\M';
         $sSQL = 'SELECT place_id FROM placex ';
         $sSQL .= 'WHERE parent_place_id in ('.$sPlaceIDs.')';
-        $sSQL .= "  AND housenumber ~* E'".$sHouseNumberRegex."'";
+        $sSQL .= "  AND transliteration(housenumber) ~* E'".$sHouseNumberRegex."'";
         $sSQL .= $this->oContext->excludeSQL(' AND place_id');
 
         Debug::printSQL($sSQL);
@@ -1019,7 +1027,7 @@ class SearchDescription
                 'Name terms (stop words)' => $this->aNameNonSearch,
                 'Address terms' => $this->aAddress,
                 'Address terms (stop words)' => $this->aAddressNonSearch,
-                'Address terms (full words)' => $this->aFullNameAddress ?? '',
+                'Address terms (full words)' => $this->aFullNameAddress,
                 'Special search' => $this->iOperator,
                 'Class' => $this->sClass,
                 'Type' => $this->sType,
@@ -1031,7 +1039,7 @@ class SearchDescription
     public function dumpAsHtmlTableRow(&$aWordIDs)
     {
         $kf = function ($k) use (&$aWordIDs) {
-            return $aWordIDs[$k] ?? '['.$k.']';
+            return $aWordIDs[$k];
         };
 
         echo '<tr>';

lib/Shell.php

@@ -7,7 +7,7 @@ class Shell
     public function __construct($sBaseCmd, ...$aParams)
     {
         if (!$sBaseCmd) {
-            throw new \Exception('Command missing in new() call');
+            throw new Exception('Command missing in new() call');
         }
         $this->baseCmd = $sBaseCmd;
         $this->aParams = array();
@@ -48,7 +48,7 @@ class Shell
         return join(' ', $aEscaped);
     }
 
-    public function run($bExitOnFail = false)
+    public function run()
     {
         $sCmd = $this->escapedCmd();
         // $aEnv does not need escaping, proc_open seems to handle it fine
@@ -67,11 +67,6 @@ class Shell
         fclose($aPipes[0]); // no stdin
 
         $iStat = proc_close($hProc);
-
-        if ($iStat != 0 && $bExitOnFail) {
-            exit($iStat);
-        }
-
         return $iStat;
     }
 

lib/Status.php

@@ -56,10 +56,4 @@ class Status
 
         return $iDataDateEpoch;
     }
-
-    public function databaseVersion()
-    {
-        $sSQL = 'SELECT value FROM nominatim_properties WHERE property = \'database_version\'';
-        return $this->oDB->getOne($sSQL);
-    }
 }

lib/TokenList.php

@@ -2,12 +2,12 @@
 
 namespace Nominatim;
 
-require_once(CONST_LibDir.'/TokenCountry.php');
-require_once(CONST_LibDir.'/TokenHousenumber.php');
-require_once(CONST_LibDir.'/TokenPostcode.php');
-require_once(CONST_LibDir.'/TokenSpecialTerm.php');
-require_once(CONST_LibDir.'/TokenWord.php');
-require_once(CONST_LibDir.'/SpecialSearchOperator.php');
+require_once(CONST_BasePath.'/lib/TokenCountry.php');
+require_once(CONST_BasePath.'/lib/TokenHousenumber.php');
+require_once(CONST_BasePath.'/lib/TokenPostcode.php');
+require_once(CONST_BasePath.'/lib/TokenSpecialTerm.php');
+require_once(CONST_BasePath.'/lib/TokenWord.php');
+require_once(CONST_BasePath.'/lib/SpecialSearchOperator.php');
 
 /**
  * Saves information about the tokens that appear in a search query.

lib/TokenSpecialTerm.php

@@ -2,7 +2,7 @@
 
 namespace Nominatim\Token;
 
-require_once(CONST_LibDir.'/SpecialSearchOperator.php');
+require_once(CONST_BasePath.'/lib/SpecialSearchOperator.php');
 
 /**
  * A word token describing a place type.

lib/cmd.php

@@ -1,6 +1,6 @@
 <?php
 
-require_once(CONST_LibDir.'/Shell.php');
+require_once(CONST_BasePath.'/lib/Shell.php');
 
 function getCmdOpt($aArg, $aSpec, &$aResult, $bExitOnError = false, $bExitOnUnknown = false)
 {
@@ -147,7 +147,7 @@ function repeatWarnings()
 function runSQLScript($sScript, $bfatal = true, $bVerbose = false, $bIgnoreErrors = false)
 {
     // Convert database DSN to psql parameters
-    $aDSNInfo = \Nominatim\DB::parseDSN(getSetting('DATABASE_DSN'));
+    $aDSNInfo = \Nominatim\DB::parseDSN(CONST_Database_DSN);
     if (!isset($aDSNInfo['port']) || !$aDSNInfo['port']) $aDSNInfo['port'] = 5432;
 
     $oCmd = new \Nominatim\Shell('psql');
@@ -195,30 +195,3 @@ function runSQLScript($sScript, $bfatal = true, $bVerbose = false, $bIgnoreError
         fail("pgsql returned with error code ($iReturn)");
     }
 }
-
-function setupHTTPProxy()
-{
-    if (!getSettingBool('HTTP_PROXY')) {
-        return;
-    }
-
-    $sProxy = 'tcp://'.getSetting('HTTP_PROXY_HOST').':'.getSetting('HTTP_PROXY_PROT');
-    $aHeaders = array();
-
-    $sLogin = getSetting('HTTP_PROXY_LOGIN');
-    $sPassword = getSetting('HTTP_PROXY_PASSWORD');
-
-    if ($sLogin && $sPassword) {
-        $sAuth = base64_encode($sLogin.':'.$sPassword);
-        $aHeaders = array('Proxy-Authorization: Basic '.$sAuth);
-    }
-
-    $aProxyHeader = array(
-                     'proxy' => $sProxy,
-                     'request_fulluri' => true,
-                     'header' => $aHeaders
-                    );
-
-    $aContext = array('http' => $aProxyHeader, 'https' => $aProxyHeader);
-    stream_context_set_default($aContext);
-}

lib/init-cmd.php

@@ -0,0 +1,28 @@
+<?php
+
+require_once('init.php');
+require_once('cmd.php');
+require_once('DebugNone.php');
+
+// handle http proxy when using file_get_contents
+if (CONST_HTTP_Proxy) {
+    $proxy = 'tcp://' . CONST_HTTP_Proxy_Host . ':' . CONST_HTTP_Proxy_Port;
+    $aHeaders = array();
+    if (CONST_HTTP_Proxy_Login != null && CONST_HTTP_Proxy_Login != '' && CONST_HTTP_Proxy_Password != null && CONST_HTTP_Proxy_Password != '') {
+        $auth = base64_encode(CONST_HTTP_Proxy_Login . ':' . CONST_HTTP_Proxy_Password);
+        $aHeaders = array("Proxy-Authorization: Basic $auth");
+    }
+    $aContext = array(
+                 'http' => array(
+                            'proxy' => $proxy,
+                            'request_fulluri' => true,
+                            'header' => $aHeaders
+                           ),
+                 'https' => array(
+                             'proxy' => $proxy,
+                             'request_fulluri' => true,
+                             'header' => $aHeaders
+                            )
+                );
+    stream_context_set_default($aContext);
+}

lib/init-website.php

@@ -20,7 +20,7 @@ function exception_handler_json($exception)
 {
     http_response_code($exception->getCode());
     header('Content-type: application/json; charset=utf-8');
-    include(CONST_LibDir.'/template/error-json.php');
+    include(CONST_BasePath.'/lib/template/error-json.php');
     exit();
 }
 
@@ -29,7 +29,7 @@ function exception_handler_xml($exception)
     http_response_code($exception->getCode());
     header('Content-type: text/xml; charset=utf-8');
     echo '<?xml version="1.0" encoding="UTF-8" ?>'."\n";
-    include(CONST_LibDir.'/template/error-xml.php');
+    include(CONST_BasePath.'/lib/template/error-xml.php');
     exit();
 }
 

lib/init.php

@@ -0,0 +1,4 @@
+<?php
+
+require_once(CONST_BasePath.'/lib/lib.php');
+require_once(CONST_BasePath.'/lib/DB.php');

lib/lib.php

@@ -1,48 +1,5 @@
 <?php
 
-function loadSettings($sProjectDir)
-{
-    @define('CONST_InstallDir', $sProjectDir);
-    // Temporary hack to set the direcory via environment instead of
-    // the installed scripts. Neither setting is part of the official
-    // set of settings.
-    defined('CONST_DataDir') or define('CONST_DataDir', $_SERVER['NOMINATIM_DATADIR']);
-    defined('CONST_SqlDir') or define('CONST_SqlDir', $_SERVER['NOMINATIM_SQLDIR']);
-    defined('CONST_ConfigDir') or define('CONST_ConfigDir', $_SERVER['NOMINATIM_CONFIGDIR']);
-    defined('CONST_Default_ModulePath') or define('CONST_Default_ModulePath', $_SERVER['NOMINATIM_DATABASE_MODULE_SRC_PATH']);
-}
-
-function getSetting($sConfName, $sDefault = null)
-{
-    $sValue = $_SERVER['NOMINATIM_'.$sConfName];
-
-    if ($sDefault !== null && !$sValue) {
-        return $sDefault;
-    }
-
-    return $sValue;
-}
-
-function getSettingBool($sConfName)
-{
-    $sVal = strtolower(getSetting($sConfName));
-
-    return strcmp($sVal, 'yes') == 0
-           || strcmp($sVal, 'true') == 0
-           || strcmp($sVal, '1') == 0;
-}
-
-function getSettingConfig($sConfName, $sSystemConfig)
-{
-    $sValue = $_SERVER['NOMINATIM_'.$sConfName];
-
-    if (!$sValue) {
-        return CONST_ConfigDir.'/'.$sSystemConfig;
-    }
-
-    return $sValue;
-}
-
 function fail($sError, $sUserError = false)
 {
     if (!$sUserError) $sUserError = $sError;
@@ -208,6 +165,17 @@ function parseLatLon($sQuery)
     return array($sFound, $fQueryLat, $fQueryLon);
 }
 
+function createPointsAroundCenter($fLon, $fLat, $fRadius)
+{
+    $iSteps = max(8, min(100, ($fRadius * 40000)^2));
+    $fStepSize = (2*pi())/$iSteps;
+    $aPolyPoints = array();
+    for ($f = 0; $f < 2*pi(); $f += $fStepSize) {
+        $aPolyPoints[] = array('', $fLon+($fRadius*sin($f)), $fLat+($fRadius*cos($f)) );
+    }
+    return $aPolyPoints;
+}
+
 function closestHouseNumber($aRow)
 {
     $fHouse = $aRow['startnumber']
@@ -227,3 +195,25 @@ function closestHouseNumber($aRow)
 
     return max(min($aRow['endnumber'], $iHn), $aRow['startnumber']);
 }
+
+function getSearchRankLabel($iRank)
+{
+    if (!isset($iRank)) return 'unknown';
+    if ($iRank < 2) return 'continent';
+    if ($iRank < 4) return 'sea';
+    if ($iRank < 8) return 'country';
+    if ($iRank < 12) return 'state';
+    if ($iRank < 16) return 'county';
+    if ($iRank == 16) return 'city';
+    if ($iRank == 17) return 'town / island';
+    if ($iRank == 18) return 'village / hamlet';
+    if ($iRank == 20) return 'suburb';
+    if ($iRank == 21) return 'postcode area';
+    if ($iRank == 22) return 'croft / farm / locality / islet';
+    if ($iRank == 23) return 'postcode area';
+    if ($iRank == 25) return 'postcode point';
+    if ($iRank == 26) return 'street / major landmark';
+    if ($iRank == 27) return 'minory street / path';
+    if ($iRank == 28) return 'house / building';
+    return 'other: ' . $iRank;
+}

lib/output.php

@@ -0,0 +1,73 @@
+<?php
+
+
+function formatOSMType($sType, $bIncludeExternal = true)
+{
+    if ($sType == 'N') return 'node';
+    if ($sType == 'W') return 'way';
+    if ($sType == 'R') return 'relation';
+
+    if (!$bIncludeExternal) return '';
+
+    if ($sType == 'T') return 'way';
+    if ($sType == 'I') return 'way';
+
+    // not handled: P, L
+
+    return '';
+}
+
+function osmLink($aFeature, $sRefText = false)
+{
+    $sOSMType = formatOSMType($aFeature['osm_type'], false);
+    if ($sOSMType) {
+        return '<a href="//www.openstreetmap.org/'.$sOSMType.'/'.$aFeature['osm_id'].'">'.$sOSMType.' '.($sRefText?$sRefText:$aFeature['osm_id']).'</a>';
+    }
+    return '';
+}
+
+function wikipediaLink($aFeature)
+{
+    if ($aFeature['wikipedia']) {
+        list($sLanguage, $sArticle) = explode(':', $aFeature['wikipedia']);
+        return '<a href="https://'.$sLanguage.'.wikipedia.org/wiki/'.urlencode($sArticle).'" target="_blank">'.$aFeature['wikipedia'].'</a>';
+    }
+    return '';
+}
+
+function detailsLink($aFeature, $sTitle = false, $sExtraProperties = false)
+{
+    if (!$aFeature['place_id']) return '';
+
+    $sHtml = '<a ';
+    if ($sExtraProperties) {
+        $sHtml .= $sExtraProperties.' ';
+    }
+
+    $sHtml .= 'href="details.php?place_id='.$aFeature['place_id'].'">'.($sTitle?$sTitle:$aFeature['place_id']).'</a>';
+
+    return $sHtml;
+}
+
+function detailsPermaLink($aFeature, $sRefText = false, $sExtraProperties = false)
+{
+    $sOSMType = formatOSMType($aFeature['osm_type'], false);
+
+    if ($sOSMType) {
+        $sHtml = '<a ';
+        if ($sExtraProperties) {
+            $sHtml .= $sExtraProperties.' ';
+        }
+        $sHtml .= 'href="details.php?osmtype='.$aFeature['osm_type']
+                  .'&osmid='.$aFeature['osm_id'].'&class='.$aFeature['class'].'">';
+
+        if ($sRefText) {
+            $sHtml .= $sRefText.'</a>';
+        } else {
+            $sHtml .= $sOSMType.' '.$aFeature['osm_id'].'</a>';
+        }
+
+        return $sHtml;
+    }
+    return detailsLink($aFeature, $sRefText, $sExtraProperties);
+}

lib/setup/AddressLevelParser.php

@@ -0,0 +1,98 @@
+<?php
+
+namespace Nominatim\Setup;
+
+/**
+ * Parses an address level description.
+ */
+class AddressLevelParser
+{
+    private $aLevels;
+
+    public function __construct($sDescriptionFile)
+    {
+        $sJson = file_get_contents($sDescriptionFile);
+        $this->aLevels = json_decode($sJson, true);
+        if (!$this->aLevels) {
+            switch (json_last_error()) {
+                case JSON_ERROR_NONE:
+                    break;
+                case JSON_ERROR_DEPTH:
+                    fail('JSON error - Maximum stack depth exceeded');
+                    break;
+                case JSON_ERROR_STATE_MISMATCH:
+                    fail('JSON error - Underflow or the modes mismatch');
+                    break;
+                case JSON_ERROR_CTRL_CHAR:
+                    fail('JSON error - Unexpected control character found');
+                    break;
+                case JSON_ERROR_SYNTAX:
+                    fail('JSON error - Syntax error, malformed JSON');
+                    break;
+                case JSON_ERROR_UTF8:
+                    fail('JSON error - Malformed UTF-8 characters, possibly incorrectly encoded');
+                    break;
+                default:
+                    fail('JSON error - Unknown error');
+                    break;
+            }
+        }
+    }
+
+    /**
+     * Dump the description into a database table.
+     *
+     * @param object $oDB    Database conneciton to use.
+     * @param string $sTable Name of table to create.
+     *
+     * @return null
+     *
+     * A new table is created. Any previously existing table is dropped.
+     * The table has the following columns:
+     * country, class, type, rank_search, rank_address.
+     */
+    public function createTable($oDB, $sTable)
+    {
+        $oDB->exec('DROP TABLE IF EXISTS '.$sTable);
+        $sSql = 'CREATE TABLE '.$sTable;
+        $sSql .= '(country_code varchar(2), class TEXT, type TEXT,';
+        $sSql .= ' rank_search SMALLINT, rank_address SMALLINT)';
+        $oDB->exec($sSql);
+
+        $sSql = 'CREATE UNIQUE INDEX ON '.$sTable.' (country_code, class, type)';
+        $oDB->exec($sSql);
+
+        $sSql = 'INSERT INTO '.$sTable.' VALUES ';
+        foreach ($this->aLevels as $aLevel) {
+            $aCountries = array();
+            if (isset($aLevel['countries'])) {
+                foreach ($aLevel['countries'] as $sCountry) {
+                    $aCountries[$sCountry] = $oDB->getDBQuoted($sCountry);
+                }
+            } else {
+                $aCountries['NULL'] = 'NULL';
+            }
+            foreach ($aLevel['tags'] as $sKey => $aValues) {
+                foreach ($aValues as $sValue => $mRanks) {
+                    $aFields = array(
+                        $oDB->getDBQuoted($sKey),
+                        $sValue ? $oDB->getDBQuoted($sValue) : 'NULL'
+                    );
+                    if (is_array($mRanks)) {
+                        $aFields[] = (string) $mRanks[0];
+                        $aFields[] = (string) $mRanks[1];
+                    } else {
+                        $aFields[] = (string) $mRanks;
+                        $aFields[] = (string) $mRanks;
+                    }
+                    $sLine = ','.join(',', $aFields).'),';
+
+                    foreach ($aCountries as $sCountries) {
+                        $sSql .= '('.$sCountries.$sLine;
+                    }
+                }
+            }
+        }
+        $oDB->exec(rtrim($sSql, ','));
+    }
+}

lib/setup/SetupClass.php

@@ -0,0 +1,942 @@
+<?php
+
+namespace Nominatim\Setup;
+
+require_once(CONST_BasePath.'/lib/setup/AddressLevelParser.php');
+require_once(CONST_BasePath.'/lib/Shell.php');
+
+class SetupFunctions
+{
+    protected $iCacheMemory;
+    protected $iInstances;
+    protected $sModulePath;
+    protected $aDSNInfo;
+    protected $bQuiet;
+    protected $bVerbose;
+    protected $sIgnoreErrors;
+    protected $bEnableDiffUpdates;
+    protected $bEnableDebugStatements;
+    protected $bNoPartitions;
+    protected $bDrop;
+    protected $oDB = null;
+
+    public function __construct(array $aCMDResult)
+    {
+        // by default, use all but one processor, but never more than 15.
+        $this->iInstances = isset($aCMDResult['threads'])
+            ? $aCMDResult['threads']
+            : (min(16, getProcessorCount()) - 1);
+
+        if ($this->iInstances < 1) {
+            $this->iInstances = 1;
+            warn('resetting threads to '.$this->iInstances);
+        }
+
+        if (isset($aCMDResult['osm2pgsql-cache'])) {
+            $this->iCacheMemory = $aCMDResult['osm2pgsql-cache'];
+        } elseif (!is_null(CONST_Osm2pgsql_Flatnode_File)) {
+            // When flatnode files are enabled then disable cache per default.
+            $this->iCacheMemory = 0;
+        } else {
+            // Otherwise: Assume we can steal all the cache memory in the box.
+            $this->iCacheMemory = getCacheMemoryMB();
+        }
+
+        $this->sModulePath = CONST_Database_Module_Path;
+        info('module path: ' . $this->sModulePath);
+
+        // parse database string
+        $this->aDSNInfo = \Nominatim\DB::parseDSN(CONST_Database_DSN);
+        if (!isset($this->aDSNInfo['port'])) {
+            $this->aDSNInfo['port'] = 5432;
+        }
+
+        // setting member variables based on command line options stored in $aCMDResult
+        $this->bQuiet = isset($aCMDResult['quiet']) && $aCMDResult['quiet'];
+        $this->bVerbose = $aCMDResult['verbose'];
+
+        //setting default values which are not set by the update.php array
+        if (isset($aCMDResult['ignore-errors'])) {
+            $this->sIgnoreErrors = $aCMDResult['ignore-errors'];
+        } else {
+            $this->sIgnoreErrors = false;
+        }
+        if (isset($aCMDResult['enable-debug-statements'])) {
+            $this->bEnableDebugStatements = $aCMDResult['enable-debug-statements'];
+        } else {
+            $this->bEnableDebugStatements = false;
+        }
+        if (isset($aCMDResult['no-partitions'])) {
+            $this->bNoPartitions = $aCMDResult['no-partitions'];
+        } else {
+            $this->bNoPartitions = false;
+        }
+        if (isset($aCMDResult['enable-diff-updates'])) {
+            $this->bEnableDiffUpdates = $aCMDResult['enable-diff-updates'];
+        } else {
+            $this->bEnableDiffUpdates = false;
+        }
+
+        $this->bDrop = isset($aCMDResult['drop']) && $aCMDResult['drop'];
+    }
+
+    public function createDB()
+    {
+        info('Create DB');
+        $oDB = new \Nominatim\DB;
+
+        if ($oDB->checkConnection()) {
+            fail('database already exists ('.CONST_Database_DSN.')');
+        }
+
+        $oCmd = (new \Nominatim\Shell('createdb'))
+                ->addParams('-E', 'UTF-8')
+                ->addParams('-p', $this->aDSNInfo['port']);
+
+        if (isset($this->aDSNInfo['username'])) {
+            $oCmd->addParams('-U', $this->aDSNInfo['username']);
+        }
+        if (isset($this->aDSNInfo['password'])) {
+            $oCmd->addEnvPair('PGPASSWORD', $this->aDSNInfo['password']);
+        }
+        if (isset($this->aDSNInfo['hostspec'])) {
+            $oCmd->addParams('-h', $this->aDSNInfo['hostspec']);
+        }
+        $oCmd->addParams($this->aDSNInfo['database']);
+
+        $result = $oCmd->run();
+        if ($result != 0) fail('Error executing external command: '.$oCmd->escapedCmd());
+    }
+
+    public function setupDB()
+    {
+        info('Setup DB');
+
+        $fPostgresVersion = $this->db()->getPostgresVersion();
+        echo 'Postgres version found: '.$fPostgresVersion."\n";
+
+        if ($fPostgresVersion < 9.03) {
+            fail('Minimum supported version of Postgresql is 9.3.');
+        }
+
+        $this->pgsqlRunScript('CREATE EXTENSION IF NOT EXISTS hstore');
+        $this->pgsqlRunScript('CREATE EXTENSION IF NOT EXISTS postgis');
+
+        $fPostgisVersion = $this->db()->getPostgisVersion();
+        echo 'Postgis version found: '.$fPostgisVersion."\n";
+
+        if ($fPostgisVersion < 2.2) {
+            echo "Minimum required Postgis version 2.2\n";
+            exit(1);
+        }
+
+        $i = $this->db()->getOne("select count(*) from pg_user where usename = '".CONST_Database_Web_User."'");
+        if ($i == 0) {
+            echo "\nERROR: Web user '".CONST_Database_Web_User."' does not exist. Create it with:\n";
+            echo "\n          createuser ".CONST_Database_Web_User."\n\n";
+            exit(1);
+        }
+
+        // Try accessing the C module, so we know early if something is wrong
+        checkModulePresence(); // raises exception on failure
+
+        if (!file_exists(CONST_ExtraDataPath.'/country_osm_grid.sql.gz')) {
+            echo 'Error: you need to download the country_osm_grid first:';
+            echo "\n    wget -O ".CONST_ExtraDataPath."/country_osm_grid.sql.gz https://www.nominatim.org/data/country_grid.sql.gz\n";
+            exit(1);
+        }
+        $this->pgsqlRunScriptFile(CONST_BasePath.'/data/country_name.sql');
+        $this->pgsqlRunScriptFile(CONST_ExtraDataPath.'/country_osm_grid.sql.gz');
+        $this->pgsqlRunScriptFile(CONST_BasePath.'/data/gb_postcode_table.sql');
+        $this->pgsqlRunScriptFile(CONST_BasePath.'/data/us_postcode_table.sql');
+
+        $sPostcodeFilename = CONST_BasePath.'/data/gb_postcode_data.sql.gz';
+        if (file_exists($sPostcodeFilename)) {
+            $this->pgsqlRunScriptFile($sPostcodeFilename);
+        } else {
+            warn('optional external GB postcode table file ('.$sPostcodeFilename.') not found. Skipping.');
+        }
+
+        $sPostcodeFilename = CONST_BasePath.'/data/us_postcode_data.sql.gz';
+        if (file_exists($sPostcodeFilename)) {
+            $this->pgsqlRunScriptFile($sPostcodeFilename);
+        } else {
+            warn('optional external US postcode table file ('.$sPostcodeFilename.') not found. Skipping.');
+        }
+
+        if ($this->bNoPartitions) {
+            $this->pgsqlRunScript('update country_name set partition = 0');
+        }
+    }
+
+    public function importData($sOSMFile)
+    {
+        info('Import data');
+
+        if (!file_exists(CONST_Osm2pgsql_Binary)) {
+            echo "Check CONST_Osm2pgsql_Binary in your local settings file.\n";
+            echo "Normally you should not need to set this manually.\n";
+            fail("osm2pgsql not found in '".CONST_Osm2pgsql_Binary."'");
+        }
+
+        $oCmd = new \Nominatim\Shell(CONST_Osm2pgsql_Binary);
+        $oCmd->addParams('--style', CONST_Import_Style);
+
+        if (!is_null(CONST_Osm2pgsql_Flatnode_File) && CONST_Osm2pgsql_Flatnode_File) {
+            $oCmd->addParams('--flat-nodes', CONST_Osm2pgsql_Flatnode_File);
+        }
+        if (CONST_Tablespace_Osm2pgsql_Data) {
+            $oCmd->addParams('--tablespace-slim-data', CONST_Tablespace_Osm2pgsql_Data);
+        }
+        if (CONST_Tablespace_Osm2pgsql_Index) {
+            $oCmd->addParams('--tablespace-slim-index', CONST_Tablespace_Osm2pgsql_Index);
+        }
+        if (CONST_Tablespace_Place_Data) {
+            $oCmd->addParams('--tablespace-main-data', CONST_Tablespace_Place_Data);
+        }
+        if (CONST_Tablespace_Place_Index) {
+            $oCmd->addParams('--tablespace-main-index', CONST_Tablespace_Place_Index);
+        }
+        $oCmd->addParams('--latlong', '--slim', '--create');
+        $oCmd->addParams('--output', 'gazetteer');
+        $oCmd->addParams('--hstore');
+        $oCmd->addParams('--number-processes', 1);
+        $oCmd->addParams('--with-forward-dependencies', 'false');
+        $oCmd->addParams('--log-progress', 'true');
+        $oCmd->addParams('--cache', $this->iCacheMemory);
+        $oCmd->addParams('--port', $this->aDSNInfo['port']);
+
+        if (isset($this->aDSNInfo['username'])) {
+            $oCmd->addParams('--username', $this->aDSNInfo['username']);
+        }
+        if (isset($this->aDSNInfo['password'])) {
+            $oCmd->addEnvPair('PGPASSWORD', $this->aDSNInfo['password']);
+        }
+        if (isset($this->aDSNInfo['hostspec'])) {
+            $oCmd->addParams('--host', $this->aDSNInfo['hostspec']);
+        }
+        $oCmd->addParams('--database', $this->aDSNInfo['database']);
+        $oCmd->addParams($sOSMFile);
+        $oCmd->run();
+
+        if (!$this->sIgnoreErrors && !$this->db()->getRow('select * from place limit 1')) {
+            fail('No Data');
+        }
+
+        if ($this->bDrop) {
+            $this->dropTable('planet_osm_nodes');
+            $this->removeFlatnodeFile();
+        }
+    }
+
+    public function createFunctions()
+    {
+        info('Create Functions');
+
+        // Try accessing the C module, so we know early if something is wrong
+        checkModulePresence(); // raises exception on failure
+
+        $this->createSqlFunctions();
+    }
+
+    public function createTables($bReverseOnly = false)
+    {
+        info('Create Tables');
+
+        $sTemplate = file_get_contents(CONST_BasePath.'/sql/tables.sql');
+        $sTemplate = $this->replaceSqlPatterns($sTemplate);
+
+        $this->pgsqlRunScript($sTemplate, false);
+
+        if ($bReverseOnly) {
+            $this->dropTable('search_name');
+        }
+
+        $oAlParser = new AddressLevelParser(CONST_Address_Level_Config);
+        $oAlParser->createTable($this->db(), 'address_levels');
+    }
+
+    public function createTableTriggers()
+    {
+        info('Create Tables');
+
+        $sTemplate = file_get_contents(CONST_BasePath.'/sql/table-triggers.sql');
+        $sTemplate = $this->replaceSqlPatterns($sTemplate);
+
+        $this->pgsqlRunScript($sTemplate, false);
+    }
+
+    public function createPartitionTables()
+    {
+        info('Create Partition Tables');
+
+        $sTemplate = file_get_contents(CONST_BasePath.'/sql/partition-tables.src.sql');
+        $sTemplate = $this->replaceSqlPatterns($sTemplate);
+
+        $this->pgsqlRunPartitionScript($sTemplate);
+    }
+
+    public function createPartitionFunctions()
+    {
+        info('Create Partition Functions');
+
+        $sTemplate = file_get_contents(CONST_BasePath.'/sql/partition-functions.src.sql');
+        $this->pgsqlRunPartitionScript($sTemplate);
+    }
+
+    public function importWikipediaArticles()
+    {
+        $sWikiArticlesFile = CONST_Wikipedia_Data_Path.'/wikimedia-importance.sql.gz';
+        if (file_exists($sWikiArticlesFile)) {
+            info('Importing wikipedia articles and redirects');
+            $this->dropTable('wikipedia_article');
+            $this->dropTable('wikipedia_redirect');
+            $this->pgsqlRunScriptFile($sWikiArticlesFile);
+        } else {
+            warn('wikipedia importance dump file not found - places will have default importance');
+        }
+    }
+
+    public function loadData($bDisableTokenPrecalc)
+    {
+        info('Drop old Data');
+
+        $oDB = $this->db();
+
+        $oDB->exec('TRUNCATE word');
+        echo '.';
+        $oDB->exec('TRUNCATE placex');
+        echo '.';
+        $oDB->exec('TRUNCATE location_property_osmline');
+        echo '.';
+        $oDB->exec('TRUNCATE place_addressline');
+        echo '.';
+        $oDB->exec('TRUNCATE location_area');
+        echo '.';
+        if (!$this->dbReverseOnly()) {
+            $oDB->exec('TRUNCATE search_name');
+            echo '.';
+        }
+        $oDB->exec('TRUNCATE search_name_blank');
+        echo '.';
+        $oDB->exec('DROP SEQUENCE seq_place');
+        echo '.';
+        $oDB->exec('CREATE SEQUENCE seq_place start 100000');
+        echo '.';
+
+        $sSQL = 'select distinct partition from country_name';
+        $aPartitions = $oDB->getCol($sSQL);
+
+        if (!$this->bNoPartitions) $aPartitions[] = 0;
+        foreach ($aPartitions as $sPartition) {
+            $oDB->exec('TRUNCATE location_road_'.$sPartition);
+            echo '.';
+        }
+
+        // used by getorcreate_word_id to ignore frequent partial words
+        $sSQL = 'CREATE OR REPLACE FUNCTION get_maxwordfreq() RETURNS integer AS ';
+        $sSQL .= '$$ SELECT '.CONST_Max_Word_Frequency.' as maxwordfreq; $$ LANGUAGE SQL IMMUTABLE';
+        $oDB->exec($sSQL);
+        echo ".\n";
+
+        // pre-create the word list
+        if (!$bDisableTokenPrecalc) {
+            info('Loading word list');
+            $this->pgsqlRunScriptFile(CONST_BasePath.'/data/words.sql');
+        }
+
+        info('Load Data');
+        $sColumns = 'osm_type, osm_id, class, type, name, admin_level, address, extratags, geometry';
+
+        $aDBInstances = array();
+        $iLoadThreads = max(1, $this->iInstances - 1);
+        for ($i = 0; $i < $iLoadThreads; $i++) {
+            // https://secure.php.net/manual/en/function.pg-connect.php
+            $DSN = CONST_Database_DSN;
+            $DSN = preg_replace('/^pgsql:/', '', $DSN);
+            $DSN = preg_replace('/;/', ' ', $DSN);
+            $aDBInstances[$i] = pg_connect($DSN, PGSQL_CONNECT_FORCE_NEW);
+            pg_ping($aDBInstances[$i]);
+        }
+
+        for ($i = 0; $i < $iLoadThreads; $i++) {
+            $sSQL = "INSERT INTO placex ($sColumns) SELECT $sColumns FROM place WHERE osm_id % $iLoadThreads = $i";
+            $sSQL .= " and not (class='place' and type='houses' and osm_type='W'";
+            $sSQL .= "          and ST_GeometryType(geometry) = 'ST_LineString')";
+            $sSQL .= ' and ST_IsValid(geometry)';
+            if ($this->bVerbose) echo "$sSQL\n";
+            if (!pg_send_query($aDBInstances[$i], $sSQL)) {
+                fail(pg_last_error($aDBInstances[$i]));
+            }
+        }
+
+        // last thread for interpolation lines
+        // https://secure.php.net/manual/en/function.pg-connect.php
+        $DSN = CONST_Database_DSN;
+        $DSN = preg_replace('/^pgsql:/', '', $DSN);
+        $DSN = preg_replace('/;/', ' ', $DSN);
+        $aDBInstances[$iLoadThreads] = pg_connect($DSN, PGSQL_CONNECT_FORCE_NEW);
+        pg_ping($aDBInstances[$iLoadThreads]);
+        $sSQL = 'insert into location_property_osmline';
+        $sSQL .= ' (osm_id, address, linegeo)';
+        $sSQL .= ' SELECT osm_id, address, geometry from place where ';
+        $sSQL .= "class='place' and type='houses' and osm_type='W' and ST_GeometryType(geometry) = 'ST_LineString'";
+        if ($this->bVerbose) echo "$sSQL\n";
+        if (!pg_send_query($aDBInstances[$iLoadThreads], $sSQL)) {
+            fail(pg_last_error($aDBInstances[$iLoadThreads]));
+        }
+
+        $bFailed = false;
+        for ($i = 0; $i <= $iLoadThreads; $i++) {
+            while (($hPGresult = pg_get_result($aDBInstances[$i])) !== false) {
+                $resultStatus = pg_result_status($hPGresult);
+                // PGSQL_EMPTY_QUERY, PGSQL_COMMAND_OK, PGSQL_TUPLES_OK,
+                // PGSQL_COPY_OUT, PGSQL_COPY_IN, PGSQL_BAD_RESPONSE,
+                // PGSQL_NONFATAL_ERROR and PGSQL_FATAL_ERROR
+                // echo 'Query result ' . $i . ' is: ' . $resultStatus . "\n";
+                if ($resultStatus != PGSQL_COMMAND_OK && $resultStatus != PGSQL_TUPLES_OK) {
+                    $resultError = pg_result_error($hPGresult);
+                    echo '-- error text ' . $i . ': ' . $resultError . "\n";
+                    $bFailed = true;
+                }
+            }
+        }
+        if ($bFailed) {
+            fail('SQL errors loading placex and/or location_property_osmline tables');
+        }
+
+        for ($i = 0; $i < $this->iInstances; $i++) {
+            pg_close($aDBInstances[$i]);
+        }
+
+        echo "\n";
+        info('Reanalysing database');
+        $this->pgsqlRunScript('ANALYSE');
+
+        $sDatabaseDate = getDatabaseDate($oDB);
+        $oDB->exec('TRUNCATE import_status');
+        if (!$sDatabaseDate) {
+            warn('could not determine database date.');
+        } else {
+            $sSQL = "INSERT INTO import_status (lastimportdate) VALUES('".$sDatabaseDate."')";
+            $oDB->exec($sSQL);
+            echo "Latest data imported from $sDatabaseDate.\n";
+        }
+    }
+
+    public function importTigerData()
+    {
+        info('Import Tiger data');
+
+        $aFilenames = glob(CONST_Tiger_Data_Path.'/*.sql');
+        info('Found '.count($aFilenames).' SQL files in path '.CONST_Tiger_Data_Path);
+        if (empty($aFilenames)) {
+            warn('Tiger data import selected but no files found in path '.CONST_Tiger_Data_Path);
+            return;
+        }
+        $sTemplate = file_get_contents(CONST_BasePath.'/sql/tiger_import_start.sql');
+        $sTemplate = $this->replaceSqlPatterns($sTemplate);
+
+        $this->pgsqlRunScript($sTemplate, false);
+
+        $aDBInstances = array();
+        for ($i = 0; $i < $this->iInstances; $i++) {
+            // https://secure.php.net/manual/en/function.pg-connect.php
+            $DSN = CONST_Database_DSN;
+            $DSN = preg_replace('/^pgsql:/', '', $DSN);
+            $DSN = preg_replace('/;/', ' ', $DSN);
+            $aDBInstances[$i] = pg_connect($DSN, PGSQL_CONNECT_FORCE_NEW | PGSQL_CONNECT_ASYNC);
+            pg_ping($aDBInstances[$i]);
+        }
+
+        foreach ($aFilenames as $sFile) {
+            echo $sFile.': ';
+            $hFile = fopen($sFile, 'r');
+            $sSQL = fgets($hFile, 100000);
+            $iLines = 0;
+            while (true) {
+                for ($i = 0; $i < $this->iInstances; $i++) {
+                    if (!pg_connection_busy($aDBInstances[$i])) {
+                        while (pg_get_result($aDBInstances[$i]));
+                        $sSQL = fgets($hFile, 100000);
+                        if (!$sSQL) break 2;
+                        if (!pg_send_query($aDBInstances[$i], $sSQL)) fail(pg_last_error($aDBInstances[$i]));
+                        $iLines++;
+                        if ($iLines == 1000) {
+                            echo '.';
+                            $iLines = 0;
+                        }
+                    }
+                }
+                usleep(10);
+            }
+            fclose($hFile);
+
+            $bAnyBusy = true;
+            while ($bAnyBusy) {
+                $bAnyBusy = false;
+                for ($i = 0; $i < $this->iInstances; $i++) {
+                    if (pg_connection_busy($aDBInstances[$i])) $bAnyBusy = true;
+                }
+                usleep(10);
+            }
+            echo "\n";
+        }
+
+        for ($i = 0; $i < $this->iInstances; $i++) {
+            pg_close($aDBInstances[$i]);
+        }
+
+        info('Creating indexes on Tiger data');
+        $sTemplate = file_get_contents(CONST_BasePath.'/sql/tiger_import_finish.sql');
+        $sTemplate = $this->replaceSqlPatterns($sTemplate);
+
+        $this->pgsqlRunScript($sTemplate, false);
+    }
+
+    public function calculatePostcodes($bCMDResultAll)
+    {
+        info('Calculate Postcodes');
+        $this->db()->exec('TRUNCATE location_postcode');
+
+        $sSQL  = 'INSERT INTO location_postcode';
+        $sSQL .= ' (place_id, indexed_status, country_code, postcode, geometry) ';
+        $sSQL .= "SELECT nextval('seq_place'), 1, country_code,";
+        $sSQL .= "       upper(trim (both ' ' from address->'postcode')) as pc,";
+        $sSQL .= '       ST_Centroid(ST_Collect(ST_Centroid(geometry)))';
+        $sSQL .= '  FROM placex';
+        $sSQL .= " WHERE address ? 'postcode' AND address->'postcode' NOT SIMILAR TO '%(,|;)%'";
+        $sSQL .= '       AND geometry IS NOT null';
+        $sSQL .= ' GROUP BY country_code, pc';
+        $this->db()->exec($sSQL);
+
+        // only add postcodes that are not yet available in OSM
+        $sSQL  = 'INSERT INTO location_postcode';
+        $sSQL .= ' (place_id, indexed_status, country_code, postcode, geometry) ';
+        $sSQL .= "SELECT nextval('seq_place'), 1, 'us', postcode,";
+        $sSQL .= '       ST_SetSRID(ST_Point(x,y),4326)';
+        $sSQL .= '  FROM us_postcode WHERE postcode NOT IN';
+        $sSQL .= '        (SELECT postcode FROM location_postcode';
+        $sSQL .= "          WHERE country_code = 'us')";
+        $this->db()->exec($sSQL);
+
+        // add missing postcodes for GB (if available)
+        $sSQL  = 'INSERT INTO location_postcode';
+        $sSQL .= ' (place_id, indexed_status, country_code, postcode, geometry) ';
+        $sSQL .= "SELECT nextval('seq_place'), 1, 'gb', postcode, geometry";
+        $sSQL .= '  FROM gb_postcode WHERE postcode NOT IN';
+        $sSQL .= '           (SELECT postcode FROM location_postcode';
+        $sSQL .= "             WHERE country_code = 'gb')";
+        $this->db()->exec($sSQL);
+
+        if (!$bCMDResultAll) {
+            $sSQL = "DELETE FROM word WHERE class='place' and type='postcode'";
+            $sSQL .= 'and word NOT IN (SELECT postcode FROM location_postcode)';
+            $this->db()->exec($sSQL);
+        }
+
+        $sSQL = 'SELECT count(getorcreate_postcode_id(v)) FROM ';
+        $sSQL .= '(SELECT distinct(postcode) as v FROM location_postcode) p';
+        $this->db()->exec($sSQL);
+    }
+
+    public function index($bIndexNoanalyse)
+    {
+        checkModulePresence(); // raises exception on failure
+
+        $oBaseCmd = (new \Nominatim\Shell(CONST_BasePath.'/nominatim/nominatim.py'))
+                    ->addParams('--database', $this->aDSNInfo['database'])
+                    ->addParams('--port', $this->aDSNInfo['port'])
+                    ->addParams('--threads', $this->iInstances);
+
+        if (!$this->bQuiet) {
+            $oBaseCmd->addParams('-v');
+        }
+        if ($this->bVerbose) {
+            $oBaseCmd->addParams('-v');
+        }
+        if (isset($this->aDSNInfo['hostspec'])) {
+            $oBaseCmd->addParams('--host', $this->aDSNInfo['hostspec']);
+        }
+        if (isset($this->aDSNInfo['username'])) {
+            $oBaseCmd->addParams('--user', $this->aDSNInfo['username']);
+        }
+        if (isset($this->aDSNInfo['password'])) {
+            $oBaseCmd->addEnvPair('PGPASSWORD', $this->aDSNInfo['password']);
+        }
+
+        info('Index ranks 0 - 4');
+        $oCmd = (clone $oBaseCmd)->addParams('--maxrank', 4);
+        echo $oCmd->escapedCmd();
+
+        $iStatus = $oCmd->run();
+        if ($iStatus != 0) {
+            fail('error status ' . $iStatus . ' running nominatim!');
+        }
+        if (!$bIndexNoanalyse) $this->pgsqlRunScript('ANALYSE');
+
+        info('Index administrative boundaries');
+        $oCmd = (clone $oBaseCmd)->addParams('-b');
+        $iStatus = $oCmd->run();
+        if ($iStatus != 0) {
+            fail('error status ' . $iStatus . ' running nominatim!');
+        }
+
+        info('Index ranks 5 - 25');
+        $oCmd = (clone $oBaseCmd)->addParams('--minrank', 5, '--maxrank', 25);
+        $iStatus = $oCmd->run();
+        if ($iStatus != 0) {
+            fail('error status ' . $iStatus . ' running nominatim!');
+        }
+
+        if (!$bIndexNoanalyse) $this->pgsqlRunScript('ANALYSE');
+
+        info('Index ranks 26 - 30');
+        $oCmd = (clone $oBaseCmd)->addParams('--minrank', 26);
+        $iStatus = $oCmd->run();
+        if ($iStatus != 0) {
+            fail('error status ' . $iStatus . ' running nominatim!');
+        }
+
+        info('Index postcodes');
+        $sSQL = 'UPDATE location_postcode SET indexed_status = 0';
+        $this->db()->exec($sSQL);
+    }
+
+    public function createSearchIndices()
+    {
+        info('Create Search indices');
+
+        $sSQL = 'SELECT relname FROM pg_class, pg_index ';
+        $sSQL .= 'WHERE pg_index.indisvalid = false AND pg_index.indexrelid = pg_class.oid';
+        $aInvalidIndices = $this->db()->getCol($sSQL);
+
+        foreach ($aInvalidIndices as $sIndexName) {
+            info("Cleaning up invalid index $sIndexName");
+            $this->db()->exec("DROP INDEX $sIndexName;");
+        }
+
+        $sTemplate = file_get_contents(CONST_BasePath.'/sql/indices.src.sql');
+        if (!$this->bDrop) {
+            $sTemplate .= file_get_contents(CONST_BasePath.'/sql/indices_updates.src.sql');
+        }
+        if (!$this->dbReverseOnly()) {
+            $sTemplate .= file_get_contents(CONST_BasePath.'/sql/indices_search.src.sql');
+        }
+        $sTemplate = $this->replaceSqlPatterns($sTemplate);
+
+        $this->pgsqlRunScript($sTemplate);
+    }
+
+    public function createCountryNames()
+    {
+        info('Create search index for default country names');
+
+        $this->pgsqlRunScript("select getorcreate_country(make_standard_name('uk'), 'gb')");
+        $this->pgsqlRunScript("select getorcreate_country(make_standard_name('united states'), 'us')");
+        $this->pgsqlRunScript('select count(*) from (select getorcreate_country(make_standard_name(country_code), country_code) from country_name where country_code is not null) as x');
+        $this->pgsqlRunScript("select count(*) from (select getorcreate_country(make_standard_name(name->'name'), country_code) from country_name where name ? 'name') as x");
+        $sSQL = 'select count(*) from (select getorcreate_country(make_standard_name(v),'
+            .'country_code) from (select country_code, skeys(name) as k, svals(name) as v from country_name) x where k ';
+        if (CONST_Languages) {
+            $sSQL .= 'in ';
+            $sDelim = '(';
+            foreach (explode(',', CONST_Languages) as $sLang) {
+                $sSQL .= $sDelim."'name:$sLang'";
+                $sDelim = ',';
+            }
+            $sSQL .= ')';
+        } else {
+            // all include all simple name tags
+            $sSQL .= "like 'name:%'";
+        }
+        $sSQL .= ') v';
+        $this->pgsqlRunScript($sSQL);
+    }
+
+    public function drop()
+    {
+        info('Drop tables only required for updates');
+
+        // The implementation is potentially a bit dangerous because it uses
+        // a positive selection of tables to keep, and deletes everything else.
+        // Including any tables that the unsuspecting user might have manually
+        // created. USE AT YOUR OWN PERIL.
+        // tables we want to keep. everything else goes.
+        $aKeepTables = array(
+                        '*columns',
+                        'import_polygon_*',
+                        'import_status',
+                        'place_addressline',
+                        'location_postcode',
+                        'location_property*',
+                        'placex',
+                        'search_name',
+                        'seq_*',
+                        'word',
+                        'query_log',
+                        'new_query_log',
+                        'spatial_ref_sys',
+                        'country_name',
+                        'place_classtype_*',
+                        'country_osm_grid'
+                       );
+
+        $aDropTables = array();
+        $aHaveTables = $this->db()->getListOfTables();
+
+        foreach ($aHaveTables as $sTable) {
+            $bFound = false;
+            foreach ($aKeepTables as $sKeep) {
+                if (fnmatch($sKeep, $sTable)) {
+                    $bFound = true;
+                    break;
+                }
+            }
+            if (!$bFound) array_push($aDropTables, $sTable);
+        }
+        foreach ($aDropTables as $sDrop) {
+            $this->dropTable($sDrop);
+        }
+
+        $this->removeFlatnodeFile();
+    }
+
+    /**
+     * Setup settings-frontend.php in the build/website directory
+     *
+     * @return null
+     */
+    public function setupWebsite()
+    {
+        $rOutputFile = fopen(CONST_InstallPath.'/settings/settings-frontend.php', 'w');
+
+        fwrite($rOutputFile, "<?php
+@define('CONST_BasePath', '".CONST_BasePath."');
+if (file_exists(getenv('NOMINATIM_SETTINGS'))) require_once(getenv('NOMINATIM_SETTINGS'));
+
+@define('CONST_Database_DSN', '".CONST_Database_DSN."');
+@define('CONST_Default_Language', ".(CONST_Default_Language ? ("'".CONST_Default_Language."'") : 'false').");
+@define('CONST_Log_DB', ".(CONST_Log_DB ? 'true' : 'false').");
+@define('CONST_Log_File', ".(CONST_Log_File ? ("'".CONST_Log_File."'")  : 'false').");
+@define('CONST_Max_Word_Frequency', '".CONST_Max_Word_Frequency."');
+@define('CONST_NoAccessControl', ".CONST_NoAccessControl.");
+@define('CONST_Places_Max_ID_count', ".CONST_Places_Max_ID_count.");
+@define('CONST_PolygonOutput_MaximumTypes', ".CONST_PolygonOutput_MaximumTypes.");
+@define('CONST_Search_AreaPolygons', ".CONST_Search_AreaPolygons.");
+@define('CONST_Search_BatchMode', ".(CONST_Search_BatchMode ? 'true' : 'false').");
+@define('CONST_Search_NameOnlySearchFrequencyThreshold', ".CONST_Search_NameOnlySearchFrequencyThreshold.");
+@define('CONST_Search_ReversePlanForAll', ".CONST_Search_ReversePlanForAll.");
+@define('CONST_Term_Normalization_Rules', \"".CONST_Term_Normalization_Rules."\");
+@define('CONST_Use_Aux_Location_data', ".(CONST_Use_Aux_Location_data ? 'true' : 'false').");
+@define('CONST_Use_US_Tiger_Data', ".(CONST_Use_US_Tiger_Data ? 'true' : 'false').");
+@define('CONST_MapIcon_URL', ".(CONST_MapIcon_URL ? ("'".CONST_MapIcon_URL."'") : 'false').');
+');
+        info(CONST_InstallPath.'/settings/settings-frontend.php has been set up successfully');
+    }
+
+    /**
+     * Return the connection to the database.
+     *
+     * @return Database object.
+     *
+     * Creates a new connection if none exists yet. Otherwise reuses the
+     * already established connection.
+     */
+    private function db()
+    {
+        if (is_null($this->oDB)) {
+            $this->oDB = new \Nominatim\DB();
+            $this->oDB->connect();
+        }
+
+        return $this->oDB;
+    }
+
+    private function removeFlatnodeFile()
+    {
+        if (!is_null(CONST_Osm2pgsql_Flatnode_File) && CONST_Osm2pgsql_Flatnode_File) {
+            if (file_exists(CONST_Osm2pgsql_Flatnode_File)) {
+                if ($this->bVerbose) echo 'Deleting '.CONST_Osm2pgsql_Flatnode_File."\n";
+                unlink(CONST_Osm2pgsql_Flatnode_File);
+            }
+        }
+    }
+
+    private function pgsqlRunScript($sScript, $bfatal = true)
+    {
+        runSQLScript(
+            $sScript,
+            $bfatal,
+            $this->bVerbose,
+            $this->sIgnoreErrors
+        );
+    }
+
+    private function createSqlFunctions()
+    {
+        $sBasePath = CONST_BasePath.'/sql/functions/';
+        $sTemplate = file_get_contents($sBasePath.'utils.sql');
+        $sTemplate .= file_get_contents($sBasePath.'normalization.sql');
+        $sTemplate .= file_get_contents($sBasePath.'ranking.sql');
+        $sTemplate .= file_get_contents($sBasePath.'importance.sql');
+        $sTemplate .= file_get_contents($sBasePath.'address_lookup.sql');
+        $sTemplate .= file_get_contents($sBasePath.'interpolation.sql');
+        if ($this->db()->tableExists('place')) {
+            $sTemplate .= file_get_contents($sBasePath.'place_triggers.sql');
+        }
+        if ($this->db()->tableExists('placex')) {
+            $sTemplate .= file_get_contents($sBasePath.'placex_triggers.sql');
+        }
+        if ($this->db()->tableExists('location_postcode')) {
+            $sTemplate .= file_get_contents($sBasePath.'postcode_triggers.sql');
+        }
+        $sTemplate = str_replace('{modulepath}', $this->sModulePath, $sTemplate);
+        if ($this->bEnableDiffUpdates) {
+            $sTemplate = str_replace('RETURN NEW; -- %DIFFUPDATES%', '--', $sTemplate);
+        }
+        if ($this->bEnableDebugStatements) {
+            $sTemplate = str_replace('--DEBUG:', '', $sTemplate);
+        }
+        if (CONST_Limit_Reindexing) {
+            $sTemplate = str_replace('--LIMIT INDEXING:', '', $sTemplate);
+        }
+        if (!CONST_Use_US_Tiger_Data) {
+            $sTemplate = str_replace('-- %NOTIGERDATA% ', '', $sTemplate);
+        }
+        if (!CONST_Use_Aux_Location_data) {
+            $sTemplate = str_replace('-- %NOAUXDATA% ', '', $sTemplate);
+        }
+
+        $sReverseOnly = $this->dbReverseOnly() ? 'true' : 'false';
+        $sTemplate = str_replace('%REVERSE-ONLY%', $sReverseOnly, $sTemplate);
+
+        $this->pgsqlRunScript($sTemplate);
+    }
+
+    private function pgsqlRunPartitionScript($sTemplate)
+    {
+        $sSQL = 'select distinct partition from country_name';
+        $aPartitions = $this->db()->getCol($sSQL);
+        if (!$this->bNoPartitions) $aPartitions[] = 0;
+
+        preg_match_all('#^-- start(.*?)^-- end#ms', $sTemplate, $aMatches, PREG_SET_ORDER);
+        foreach ($aMatches as $aMatch) {
+            $sResult = '';
+            foreach ($aPartitions as $sPartitionName) {
+                $sResult .= str_replace('-partition-', $sPartitionName, $aMatch[1]);
+            }
+            $sTemplate = str_replace($aMatch[0], $sResult, $sTemplate);
+        }
+
+        $this->pgsqlRunScript($sTemplate);
+    }
+
+    private function pgsqlRunScriptFile($sFilename)
+    {
+        if (!file_exists($sFilename)) fail('unable to find '.$sFilename);
+
+        $oCmd = (new \Nominatim\Shell('psql'))
+                ->addParams('--port', $this->aDSNInfo['port'])
+                ->addParams('--dbname', $this->aDSNInfo['database']);
+
+        if (!$this->bVerbose) {
+            $oCmd->addParams('--quiet');
+        }
+        if (isset($this->aDSNInfo['hostspec'])) {
+            $oCmd->addParams('--host', $this->aDSNInfo['hostspec']);
+        }
+        if (isset($this->aDSNInfo['username'])) {
+            $oCmd->addParams('--username', $this->aDSNInfo['username']);
+        }
+        if (isset($this->aDSNInfo['password'])) {
+            $oCmd->addEnvPair('PGPASSWORD', $this->aDSNInfo['password']);
+        }
+        $ahGzipPipes = null;
+        if (preg_match('/\\.gz$/', $sFilename)) {
+            $aDescriptors = array(
+                             0 => array('pipe', 'r'),
+                             1 => array('pipe', 'w'),
+                             2 => array('file', '/dev/null', 'a')
+                            );
+            $oZcatCmd = new \Nominatim\Shell('zcat', $sFilename);
+
+            $hGzipProcess = proc_open($oZcatCmd->escapedCmd(), $aDescriptors, $ahGzipPipes);
+            if (!is_resource($hGzipProcess)) fail('unable to start zcat');
+            $aReadPipe = $ahGzipPipes[1];
+            fclose($ahGzipPipes[0]);
+        } else {
+            $oCmd->addParams('--file', $sFilename);
+            $aReadPipe = array('pipe', 'r');
+        }
+        $aDescriptors = array(
+                         0 => $aReadPipe,
+                         1 => array('pipe', 'w'),
+                         2 => array('file', '/dev/null', 'a')
+                        );
+        $ahPipes = null;
+
+        $hProcess = proc_open($oCmd->escapedCmd(), $aDescriptors, $ahPipes, null, $oCmd->aEnv);
+        if (!is_resource($hProcess)) fail('unable to start pgsql');
+        // TODO: error checking
+        while (!feof($ahPipes[1])) {
+            echo fread($ahPipes[1], 4096);
+        }
+        fclose($ahPipes[1]);
+        $iReturn = proc_close($hProcess);
+        if ($iReturn > 0) {
+            fail("pgsql returned with error code ($iReturn)");
+        }
+        if ($ahGzipPipes) {
+            fclose($ahGzipPipes[1]);
+            proc_close($hGzipProcess);
+        }
+    }
+
+    private function replaceSqlPatterns($sSql)
+    {
+        $sSql = str_replace('{www-user}', CONST_Database_Web_User, $sSql);
+
+        $aPatterns = array(
+                      '{ts:address-data}' => CONST_Tablespace_Address_Data,
+                      '{ts:address-index}' => CONST_Tablespace_Address_Index,
+                      '{ts:search-data}' => CONST_Tablespace_Search_Data,
+                      '{ts:search-index}' =>  CONST_Tablespace_Search_Index,
+                      '{ts:aux-data}' =>  CONST_Tablespace_Aux_Data,
+                      '{ts:aux-index}' =>  CONST_Tablespace_Aux_Index,
+        );
+
+        foreach ($aPatterns as $sPattern => $sTablespace) {
+            if ($sTablespace) {
+                $sSql = str_replace($sPattern, 'TABLESPACE "'.$sTablespace.'"', $sSql);
+            } else {
+                $sSql = str_replace($sPattern, '', $sSql);
+            }
+        }
+
+        return $sSql;
+    }
+
+    /**
+     * Drop table with the given name if it exists.
+     *
+     * @param string $sName Name of table to remove.
+     *
+     * @return null
+     */
+    private function dropTable($sName)
+    {
+        if ($this->bVerbose) echo "Dropping table $sName\n";
+        $this->db()->deleteTable($sName);
+    }
+
+    /**
+     * Check if the database is in reverse-only mode.
+     *
+     * @return True if there is no search_name table and infrastructure.
+     */
+    private function dbReverseOnly()
+    {
+        return !($this->db()->tableExists('search_name'));
+    }
+}

lib/setup_functions.php

@@ -0,0 +1,31 @@
+<?php
+
+function checkInFile($sOSMFile)
+{
+    if (!isset($sOSMFile)) {
+        fail('missing --osm-file for data import');
+    }
+
+    if (!file_exists($sOSMFile)) {
+        fail('the path supplied to --osm-file does not exist');
+    }
+
+    if (!is_readable($sOSMFile)) {
+        fail('osm-file "' . $aCMDResult['osm-file'] . '" not readable');
+    }
+}
+
+function checkModulePresence()
+{
+    // Try accessing the C module, so we know early if something is wrong.
+    // Raises Nominatim\DatabaseError on failure
+
+    $sModulePath = CONST_Database_Module_Path;
+    $sSQL = "CREATE FUNCTION nominatim_test_import_func(text) RETURNS text AS '";
+    $sSQL .= $sModulePath . "/nominatim.so', 'transliteration' LANGUAGE c IMMUTABLE STRICT";
+    $sSQL .= ';DROP FUNCTION nominatim_test_import_func(text);';
+
+    $oDB = new \Nominatim\DB();
+    $oDB->connect();
+    $oDB->exec($sSQL, null, 'Database server failed to load '.$sModulePath.'/nominatim.so module');
+}

manual/CMakeLists.txt

@@ -1,15 +0,0 @@
-# Creates and installs manual page
-
-configure_file(${PROJECT_SOURCE_DIR}/manual/create-manpage.tmpl create_manpage.py)
-
-find_program(ARGPARSEMANPAGE argparse-manpage)
-
-ADD_CUSTOM_TARGET(manpage
-    COMMAND ${ARGPARSEMANPAGE} --pyfile ${CMAKE_CURRENT_BINARY_DIR}/create_manpage.py
-    --function get_parser --project-name Nominatim
-    --url https://nominatim.org  > ${CMAKE_CURRENT_SOURCE_DIR}/nominatim.1
-
-    COMMAND sed -i '/.SH AUTHORS/I,+2 d' ${CMAKE_CURRENT_SOURCE_DIR}/nominatim.1
-)
-
-install(FILES ${CMAKE_CURRENT_SOURCE_DIR}/nominatim.1 DESTINATION share/man/man1 )

manual/create-manpage.tmpl

@@ -1,12 +0,0 @@
-#!/usr/bin/env python3
-import sys
-import os
-
-sys.path.append('@PROJECT_SOURCE_DIR@')
-
-from nominatim.cli import get_set_parser
-
-def get_parser():
-    parser = get_set_parser(phpcgi_path='@PHPCGI_BIN@')
-
-    return parser.parser

nominatim/cli.py

@@ -1,288 +0,0 @@
-"""
-Command-line interface to the Nominatim functions for import, update,
-database administration and querying.
-"""
-import logging
-import os
-import sys
-import argparse
-from pathlib import Path
-
-from .config import Configuration
-from .tools.exec_utils import run_legacy_script, run_php_server
-from .errors import UsageError
-from . import clicmd
-from .clicmd.args import NominatimArgs
-from .tools import tiger_data
-
-LOG = logging.getLogger()
-
-
-class CommandlineParser:
-    """ Wraps some of the common functions for parsing the command line
-        and setting up subcommands.
-    """
-    def __init__(self, prog, description):
-        self.parser = argparse.ArgumentParser(
-            prog=prog,
-            description=description,
-            formatter_class=argparse.RawDescriptionHelpFormatter)
-
-        self.subs = self.parser.add_subparsers(title='available commands',
-                                               dest='subcommand')
-
-        # Arguments added to every sub-command
-        self.default_args = argparse.ArgumentParser(add_help=False)
-        group = self.default_args.add_argument_group('Default arguments')
-        group.add_argument('-h', '--help', action='help',
-                           help='Show this help message and exit')
-        group.add_argument('-q', '--quiet', action='store_const', const=0,
-                           dest='verbose', default=1,
-                           help='Print only error messages')
-        group.add_argument('-v', '--verbose', action='count', default=1,
-                           help='Increase verboseness of output')
-        group.add_argument('--project-dir', metavar='DIR', default='.',
-                           help='Base directory of the Nominatim installation (default:.)')
-        group.add_argument('-j', '--threads', metavar='NUM', type=int,
-                           help='Number of parallel threads to use')
-
-
-    def add_subcommand(self, name, cmd):
-        """ Add a subcommand to the parser. The subcommand must be a class
-            with a function add_args() that adds the parameters for the
-            subcommand and a run() function that executes the command.
-        """
-        parser = self.subs.add_parser(name, parents=[self.default_args],
-                                      help=cmd.__doc__.split('\n', 1)[0],
-                                      description=cmd.__doc__,
-                                      formatter_class=argparse.RawDescriptionHelpFormatter,
-                                      add_help=False)
-        parser.set_defaults(command=cmd)
-        cmd.add_args(parser)
-
-    def run(self, **kwargs):
-        """ Parse the command line arguments of the program and execute the
-            appropriate subcommand.
-        """
-        args = NominatimArgs()
-        self.parser.parse_args(args=kwargs.get('cli_args'), namespace=args)
-
-        if args.subcommand is None:
-            self.parser.print_help()
-            return 1
-
-        for arg in ('module_dir', 'osm2pgsql_path', 'phplib_dir', 'sqllib_dir',
-                    'data_dir', 'config_dir', 'phpcgi_path'):
-            setattr(args, arg, Path(kwargs[arg]))
-        args.project_dir = Path(args.project_dir).resolve()
-
-        if 'cli_args' not in kwargs:
-            logging.basicConfig(stream=sys.stderr,
-                                format='%(asctime)s: %(message)s',
-                                datefmt='%Y-%m-%d %H:%M:%S',
-                                level=max(4 - args.verbose, 1) * 10)
-
-        args.config = Configuration(args.project_dir, args.config_dir,
-                                    environ=kwargs.get('environ', os.environ))
-
-        log = logging.getLogger()
-        log.warning('Using project directory: %s', str(args.project_dir))
-
-        try:
-            return args.command.run(args)
-        except UsageError as exception:
-            if log.isEnabledFor(logging.DEBUG):
-                raise # use Python's exception printing
-            log.fatal('FATAL: %s', exception)
-
-        # If we get here, then execution has failed in some way.
-        return 1
-
-
-##### Subcommand classes
-#
-# Each class needs to implement two functions: add_args() adds the CLI parameters
-# for the subfunction, run() executes the subcommand.
-#
-# The class documentation doubles as the help text for the command. The
-# first line is also used in the summary when calling the program without
-# a subcommand.
-#
-# No need to document the functions each time.
-# pylint: disable=C0111
-# Using non-top-level imports to make pyosmium optional for replication only.
-# pylint: disable=E0012,C0415
-class UpdateAddData:
-    """\
-    Add additional data from a file or an online source.
-
-    Data is only imported, not indexed. You need to call `nominatim-update index`
-    to complete the process.
-    """
-
-    @staticmethod
-    def add_args(parser):
-        group_name = parser.add_argument_group('Source')
-        group = group_name.add_mutually_exclusive_group(required=True)
-        group.add_argument('--file', metavar='FILE',
-                           help='Import data from an OSM file')
-        group.add_argument('--diff', metavar='FILE',
-                           help='Import data from an OSM diff file')
-        group.add_argument('--node', metavar='ID', type=int,
-                           help='Import a single node from the API')
-        group.add_argument('--way', metavar='ID', type=int,
-                           help='Import a single way from the API')
-        group.add_argument('--relation', metavar='ID', type=int,
-                           help='Import a single relation from the API')
-        group.add_argument('--tiger-data', metavar='DIR',
-                           help='Add housenumbers from the US TIGER census database.')
-        group = parser.add_argument_group('Extra arguments')
-        group.add_argument('--use-main-api', action='store_true',
-                           help='Use OSM API instead of Overpass to download objects')
-
-    @staticmethod
-    def run(args):
-        if args.tiger_data:
-            return tiger_data.add_tiger_data(args.config.get_libpq_dsn(),
-                                             args.tiger_data,
-                                             args.threads or 1,
-                                             args.config,
-                                             args.sqllib_dir)
-
-        params = ['update.php']
-        if args.file:
-            params.extend(('--import-file', args.file))
-        elif args.diff:
-            params.extend(('--import-diff', args.diff))
-        elif args.node:
-            params.extend(('--import-node', args.node))
-        elif args.way:
-            params.extend(('--import-way', args.way))
-        elif args.relation:
-            params.extend(('--import-relation', args.relation))
-        if args.use_main_api:
-            params.append('--use-main-api')
-        return run_legacy_script(*params, nominatim_env=args)
-
-
-class QueryExport:
-    """\
-    Export addresses as CSV file from the database.
-    """
-
-    @staticmethod
-    def add_args(parser):
-        group = parser.add_argument_group('Output arguments')
-        group.add_argument('--output-type', default='street',
-                           choices=('continent', 'country', 'state', 'county',
-                                    'city', 'suburb', 'street', 'path'),
-                           help='Type of places to output (default: street)')
-        group.add_argument('--output-format',
-                           default='street;suburb;city;county;state;country',
-                           help=("Semicolon-separated list of address types "
-                                 "(see --output-type). Multiple ranks can be "
-                                 "merged into one column by simply using a "
-                                 "comma-separated list."))
-        group.add_argument('--output-all-postcodes', action='store_true',
-                           help=("List all postcodes for address instead of "
-                                 "just the most likely one"))
-        group.add_argument('--language',
-                           help=("Preferred language for output "
-                                 "(use local name, if omitted)"))
-        group = parser.add_argument_group('Filter arguments')
-        group.add_argument('--restrict-to-country', metavar='COUNTRY_CODE',
-                           help='Export only objects within country')
-        group.add_argument('--restrict-to-osm-node', metavar='ID', type=int,
-                           help='Export only children of this OSM node')
-        group.add_argument('--restrict-to-osm-way', metavar='ID', type=int,
-                           help='Export only children of this OSM way')
-        group.add_argument('--restrict-to-osm-relation', metavar='ID', type=int,
-                           help='Export only children of this OSM relation')
-
-
-    @staticmethod
-    def run(args):
-        params = ['export.php',
-                  '--output-type', args.output_type,
-                  '--output-format', args.output_format]
-        if args.output_all_postcodes:
-            params.append('--output-all-postcodes')
-        if args.language:
-            params.extend(('--language', args.language))
-        if args.restrict_to_country:
-            params.extend(('--restrict-to-country', args.restrict_to_country))
-        if args.restrict_to_osm_node:
-            params.extend(('--restrict-to-osm-node', args.restrict_to_osm_node))
-        if args.restrict_to_osm_way:
-            params.extend(('--restrict-to-osm-way', args.restrict_to_osm_way))
-        if args.restrict_to_osm_relation:
-            params.extend(('--restrict-to-osm-relation', args.restrict_to_osm_relation))
-
-        return run_legacy_script(*params, nominatim_env=args)
-
-
-class AdminServe:
-    """\
-    Start a simple web server for serving the API.
-
-    This command starts the built-in PHP webserver to serve the website
-    from the current project directory. This webserver is only suitable
-    for testing and develop. Do not use it in production setups!
-
-    By the default, the webserver can be accessed at: http://127.0.0.1:8088
-    """
-
-    @staticmethod
-    def add_args(parser):
-        group = parser.add_argument_group('Server arguments')
-        group.add_argument('--server', default='127.0.0.1:8088',
-                           help='The address the server will listen to.')
-
-    @staticmethod
-    def run(args):
-        run_php_server(args.server, args.project_dir / 'website')
-
-def get_set_parser(**kwargs):
-    """\
-    Initializes the parser and adds various subcommands for
-    nominatim cli.
-    """
-    parser = CommandlineParser('nominatim', nominatim.__doc__)
-
-    parser.add_subcommand('import', clicmd.SetupAll)
-    parser.add_subcommand('freeze', clicmd.SetupFreeze)
-    parser.add_subcommand('replication', clicmd.UpdateReplication)
-
-    parser.add_subcommand('special-phrases', clicmd.ImportSpecialPhrases)
-
-    parser.add_subcommand('add-data', UpdateAddData)
-    parser.add_subcommand('index', clicmd.UpdateIndex)
-    parser.add_subcommand('refresh', clicmd.UpdateRefresh)
-
-    parser.add_subcommand('admin', clicmd.AdminFuncs)
-
-    parser.add_subcommand('export', QueryExport)
-    parser.add_subcommand('serve', AdminServe)
-
-    if kwargs.get('phpcgi_path'):
-        parser.add_subcommand('search', clicmd.APISearch)
-        parser.add_subcommand('reverse', clicmd.APIReverse)
-        parser.add_subcommand('lookup', clicmd.APILookup)
-        parser.add_subcommand('details', clicmd.APIDetails)
-        parser.add_subcommand('status', clicmd.APIStatus)
-    else:
-        parser.parser.epilog = 'php-cgi not found. Query commands not available.'
-
-    parser.add_subcommand('transition', clicmd.AdminTransition)
-
-    return parser
-
-
-def nominatim(**kwargs):
-    """\
-    Command-line tools for importing, updating, administrating and
-    querying the Nominatim database.
-    """
-    parser = get_set_parser(**kwargs)
-
-    return parser.run(**kwargs)