prepare 3.5.2 release

Using both is slightly problematic because they have different
ways to use the index. Newer versions of Postgis exhibit a
query planner issue when both functions appear together.
As ST_Intersects includes ST_Covers, simply remove the latter.

CMakeLists.txt

@@ -20,7 +20,7 @@ project(nominatim)
 
 set(NOMINATIM_VERSION_MAJOR 3)
 set(NOMINATIM_VERSION_MINOR 5)
-set(NOMINATIM_VERSION_PATCH 0)
+set(NOMINATIM_VERSION_PATCH 2)
 
 set(NOMINATIM_VERSION "${NOMINATIM_VERSION_MAJOR}.${NOMINATIM_VERSION_MINOR}.${NOMINATIM_VERSION_PATCH}")
 

ChangeLog

@@ -1,3 +1,22 @@
+3.5.2
+
+ * ensure that wikipedia tags are imported for all styles
+ * reinstate verbosity for indexing during updates
+ * make house number reappear in display name on named POIs
+ * introduce batch processing in indexer to avoid transaction ID overrun
+ * increase splitting for large geometries to improve indexing speed
+ * remove deprecated get_magic_quotes_gpc() function
+ * make sure that all postcodes have an entry in word and are thus searchable
+ * remove use of ST_Covers in conjunction woth ST_Intersects,
+   causes bad query planning and slow updates in Postgis3
+ * update osm2pgsql
+
+3.5.1
+
+ * disable jit and parallel processing in PostgreSQL for osm2pgsql
+ * update libosmium to 2.15.6 (fixes an issue with processing hanging
+   on large multipolygons)
+
 3.5.0
 
  * structured select on HTML search page

docs/admin/Advanced-Installations.md

@@ -63,7 +63,7 @@ Run the following command from your Nominatim directory after configuring the fi
         This file uses osmium-tool. It must be installed before executing the import script.
         Installation instructions can be found [here](https://osmcode.org/osmium-tool/manual.html#installation).
 
-### Updating multiple regions
+## Updating multiple regions
 
 To import multiple regions in your database, you need to configure and run ```utils/update_database.sh```.
 This uses the update directory set up while setting up the DB.   
@@ -103,69 +103,7 @@ Run the following command from your Nominatim directory after configuring the fi
 
 This will get diffs from the replication server, import diffs and index the database. The default replication server in the script([Geofabrik](https://download.geofabrik.de)) provides daily updates.
 
-## Importing Nominatim to an external PostgreSQL database
+## Verification and further setup
 
-You can install Nominatim using a database that runs on a different server when
-you have physical access to the file system on the other server. Nominatim
-uses a custom normalization library that needs to be made accessible to the
-PostgreSQL server. This section explains how to set up the normalization
-library.
+Instructions for import verification and other details like importing Wikidata can be found in [import and update page](Import-and-Update.md)
 
-### Option 1: Compiling the library on the database server
-
-The most sure way to get a working library is to compile it on the database
-server. From the prerequisites you need at least cmake, gcc and the
-PostgreSQL server package.
-
-Clone or unpack the Nominatim source code, enter the source directory and
-create and enter a build directory.
-
-```sh
-cd Nominatim
-mkdir build
-cd build
-```
-
-Now configure cmake to only build the PostgreSQL module and build it:
-
-```
-cmake -DBUILD_IMPORTER=off -DBUILD_API=off -DBUILD_TESTS=off -DBUILD_DOCS=off -DBUILD_OSM2PGSQL=off ..
-make
-```
-
-When done, you find the normalization library in `build/module/nominatim.so`.
-Copy it to a place where it is readable and executable by the PostgreSQL server
-process.
-
-### Option 2: Compiling the library on the import machine
-
-You can also compile the normalization library on the machine from where you
-run the import.
-
-!!! important
-    You can only do this when the database server and the import machine have
-    the same architecture and run the same version of Linux. Otherwise there is
-    no guarantee that the compiled library is compatible with the PostgreSQL
-    server running on the database server.
-
-Make sure that the PostgreSQL server package is installed on the machine
-**with the same version as on the database server**. You do not need to install
-the PostgreSQL server itself.
-
-Download and compile Nominatim as per standard instructions. Once done, you find
-the nomrmalization library in `build/module/nominatim.so`. Copy the file to
-the database server at a location where it is readable and executable by the
-PostgreSQL server process.
-
-### Running the import
-
-On the client side you now need to configure the import to point to the
-correct location of the library **on the database server**. Add the following
-line to your your `settings/local.php` file:
-
-```php
-@define('CONST_Database_Module_Path', '<directory on the database server where nominatim.so resides>');
-```
-
-Now change the `CONST_Database_DSN` to point to your remote server and continue
-to follow the [standard instructions for importing](/admin/Import).

docs/admin/Faq.md

@@ -22,21 +22,6 @@ then you can resume with the following command:
 If the reported rank is 26 or higher, you can also safely add `--index-noanalyse`.
 
 
-### PostgreSQL crashed "invalid page in block"
-
-Usually serious problem, can be a hardware issue, not all data written to disc
-for example. Check PostgreSQL log file and search PostgreSQL issues/mailing
-list for hints.
-
-If it happened during index creation you can try rerunning the step with
-
-```sh
-./utils/setup.php --create-search-indices --ignore-errors
-```
-
-Otherwise it's best to start the full setup from the beginning.
-
-
 ### PHP "open_basedir restriction in effect" warnings
 
     PHP Warning:  file_get_contents(): open_basedir restriction in effect.
@@ -53,7 +38,7 @@ by adding ";" at the beginning of the line. Don't forget to enable this setting
 again once you are done with the PHP command line operations.
 
 
-### PHP timezeone warnings
+### PHP timzeone warnings
 
 The Apache log may contain lots of PHP warnings like this:
     `PHP Warning:  date_default_timezone_set() function.`
@@ -171,8 +156,7 @@ Example error message
 
 The PostgreSQL database, i.e. user `postgres`, needs to have access to that file.
 
-The permission need to be read & executable by everybody, but not writeable
-by everybody, e.g.
+The permission need to be read & executable by everybody, e.g.
 
 ```
    -rwxr-xr-x 1 nominatim nominatim 297984 build/module/nominatim.so
@@ -183,16 +167,6 @@ Try `chmod a+r nominatim.so; chmod a+x nominatim.so`.
 When running SELinux, make sure that the
 [context is set up correctly](../appendix/Install-on-Centos-7/#adding-selinux-security-settings).
 
-When you recently updated your operating system, updated PostgreSQL to
-a new version or moved files (e.g. the build directory) you should
-recreate `nominatim.so`. Try
-
-```
-   cd build
-   rm -r module/
-   cmake $main_Nominatim_path && make
-```
-
 ### Setup.php fails with "DB Error: extension not found"
 
 Make sure you have the PostgreSQL extensions "hstore" and "postgis" installed.

docs/admin/Import-and-Update.md

@@ -1,4 +1,4 @@
-# Importing the Database
+# Importing and Updating the Database
 
 The following instructions explain how to create a Nominatim database
 from an OSM planet file and how to keep the database up to date. It
@@ -253,3 +253,70 @@ entire US adds about 10GB to your database.
 ```
 
 
+## Updates
+
+There are many different ways to update your Nominatim database.
+The following section describes how to keep it up-to-date with Pyosmium.
+For a list of other methods see the output of `./utils/update.php --help`.
+
+!!! warning
+    If you have configured a flatnode file for the import, then you
+    need to keep this flatnode file around for updates as well.
+
+#### Installing the newest version of Pyosmium
+
+It is recommended to install Pyosmium via pip. Make sure to use python3.
+Run (as the same user who will later run the updates):
+
+```sh
+pip3 install --user osmium
+```
+
+Nominatim needs a tool called `pyosmium-get-updates` which comes with
+Pyosmium. You need to tell Nominatim where to find it. Add the
+following line to your `settings/local.php`:
+
+    @define('CONST_Pyosmium_Binary', '/home/user/.local/bin/pyosmium-get-changes');
+
+The path above is fine if you used the `--user` parameter with pip.
+Replace `user` with your user name.
+
+#### Setting up the update process
+
+Next the update needs to be initialised. By default Nominatim is configured
+to update using the global minutely diffs.
+
+If you want a different update source you will need to add some settings
+to `settings/local.php`. For example, to use the daily country extracts
+diffs for Ireland from Geofabrik add the following:
+
+    // base URL of the replication service
+    @define('CONST_Replication_Url', 'https://download.geofabrik.de/europe/ireland-and-northern-ireland-updates');
+    // How often upstream publishes diffs
+    @define('CONST_Replication_Update_Interval', '86400');
+    // How long to sleep if no update found yet
+    @define('CONST_Replication_Recheck_Interval', '900');
+
+To set up the update process now run the following command:
+
+    ./utils/update.php --init-updates
+
+It outputs the date where updates will start. Recheck that this date is
+what you expect.
+
+The `--init-updates` command needs to be rerun whenever the replication service
+is changed.
+
+#### Updating Nominatim
+
+The following command will keep your database constantly up to date:
+
+    ./utils/update.php --import-osmosis-all
+
+(Note that even though the old name "import-osmosis-all" has been kept for
+compatibility reasons, Osmosis is not required to run this - it uses pyosmium
+behind the scenes.)
+
+If you have imported multiple country extracts and want to keep them
+up-to-date, [Advanced installations section](Advanced-Installations.md) contains instructions 
+to set up and update multiple country extracts.

docs/admin/Installation.md

@@ -37,9 +37,9 @@ For compiling:
 For running Nominatim:
 
   * [PostgreSQL](https://www.postgresql.org) (9.3+)
-  * [PostGIS](https://postgis.net) (2.2+)
+  * [PostGIS](https://postgis.org) (2.2+)
   * [Python 3](https://www.python.org/)
-  * [Psycopg2](https://www.psycopg.org)
+  * [Psycopg2](https://initd.org/psycopg)
   * [PHP](https://php.net) (7.0 or later)
   * PHP-pgsql
   * PHP-intl (bundled with PHP)
@@ -51,7 +51,7 @@ For running continuous updates:
 
 For running tests:
 
-  * [behave](https://behave.readthedocs.io)
+  * [behave](http://pythonhosted.org/behave/)
   * [nose](https://nose.readthedocs.io)
   * [phpunit](https://phpunit.de) >= 7.3
 
@@ -188,4 +188,4 @@ Restart the nginx and php5-fpm services and the website should now be available
 at `http://localhost/`.
 
 
-Now continue with [importing the database](Import.md).
+Now continue with [importing the database](Import-and-Update.md).

docs/admin/Migration.md

@@ -17,28 +17,10 @@ follows:
   * download the new Wikipedia tables as described in the import section
   * reimport the tables: `./utils/setup.php --import-wikipedia-articles`
   * update the functions: `./utils/setup.php --create-functions --enable-diff-updates`
-  * create a new lookup index:
-```
-CREATE INDEX idx_placex_wikidata on placex
-USING BTREE ((extratags -> 'wikidata'))
-WHERE extratags ? 'wikidata' and class = 'place' and osm_type = 'N' and rank_search < 26
-```
   * compute importance: `./utils/update.php --recompute-importance`
 
 The last step takes about 10 hours on the full planet.
 
-Remove one function (it will be recreated in the next step):
-
-```sql
-DROP FUNCTION create_country(hstore,character varying);
-```
-
-Finally, update all SQL functions:
-
-```sh
-./utils/setup.php --create-functions --enable-diff-updates --create-partition-functions
-```
-
 ## 3.3.0 -> 3.4.0
 
 ### Reorganisation of location_area_country table
@@ -56,12 +38,6 @@ CREATE INDEX idx_location_area_country_geometry ON location_area_country USING G
 CREATE INDEX idx_location_area_country_place_id ON location_area_country USING BTREE (place_id);
 ```
 
-Finally, update all SQL functions:
-
-```sh
-./utils/setup.php --create-functions --enable-diff-updates --create-partition-functions
-```
-
 ## 3.2.0 -> 3.3.0
 
 ### New database connection string (DSN) format

docs/admin/Update.md

@@ -1,67 +0,0 @@
-# Updating the Database
-
-There are many different ways to update your Nominatim database.
-The following section describes how to keep it up-to-date with Pyosmium.
-For a list of other methods see the output of `./utils/update.php --help`.
-
-!!! warning
-    If you have configured a flatnode file for the import, then you
-    need to keep this flatnode file around for updates as well.
-
-#### Installing the newest version of Pyosmium
-
-It is recommended to install Pyosmium via pip. Make sure to use python3.
-Run (as the same user who will later run the updates):
-
-```sh
-pip3 install --user osmium
-```
-
-Nominatim needs a tool called `pyosmium-get-changes` which comes with
-Pyosmium. You need to tell Nominatim where to find it. Add the
-following line to your `settings/local.php`:
-
-    @define('CONST_Pyosmium_Binary', '/home/user/.local/bin/pyosmium-get-changes');
-
-The path above is fine if you used the `--user` parameter with pip.
-Replace `user` with your user name.
-
-#### Setting up the update process
-
-Next the update needs to be initialised. By default Nominatim is configured
-to update using the global minutely diffs.
-
-If you want a different update source you will need to add some settings
-to `settings/local.php`. For example, to use the daily country extracts
-diffs for Ireland from Geofabrik add the following:
-
-    // base URL of the replication service
-    @define('CONST_Replication_Url', 'https://download.geofabrik.de/europe/ireland-and-northern-ireland-updates');
-    // How often upstream publishes diffs
-    @define('CONST_Replication_Update_Interval', '86400');
-    // How long to sleep if no update found yet
-    @define('CONST_Replication_Recheck_Interval', '900');
-
-To set up the update process now run the following command:
-
-    ./utils/update.php --init-updates
-
-It outputs the date where updates will start. Recheck that this date is
-what you expect.
-
-The `--init-updates` command needs to be rerun whenever the replication service
-is changed.
-
-#### Updating Nominatim
-
-The following command will keep your database constantly up to date:
-
-    ./utils/update.php --import-osmosis-all
-
-(Note that even though the old name "import-osmosis-all" has been kept for
-compatibility reasons, Osmosis is not required to run this - it uses pyosmium
-behind the scenes.)
-
-If you have imported multiple country extracts and want to keep them
-up-to-date, [Advanced installations section](Advanced-Installations.md) contains instructions 
-to set up and update multiple country extracts.

docs/develop/Documentation.md

@@ -8,9 +8,6 @@ To preview local changes, first install MkDocs
 pip3 install --user mkdocs
 ```
 
-If `mkdocs` can't be found after the installation, the $PATH might have not
-been set correctly yet. Try opening a new terminal session.
-
 
 Then go to the build directory and run
 

docs/develop/Setup.md

@@ -1,46 +0,0 @@
-# Setup Test Environment
-
-To test changes and contribute to Nominatim you should be able to run
-the test suite(s). For many usecases it's enough to create a Vagrant
-virtual machine (see `VAGRANT.md`), import one small country into the
-database.
-
-## Prerequisites
-
-Nominatim supports a range of PHP versions and PHPUnit versions also
-move fast. We try to test against the newest stable PHP release and
-PHPUnit version even though we expect many Nominatim users will install
-older version on their production servers.
-
-#### Ubuntu 20
-
-    sudo apt-get install -y phpunit php-codesniffer php-cgi
-    pip3 install --user behave nose
-
-#### Ubuntu 18
-
-    pip3 install --user behave nose
-
-    sudo apt-get install -y composer php-cgi php-cli php-mbstring php-xml zip unzip
-
-    composer global require "squizlabs/php_codesniffer=*"
-    sudo ln -s ~/.config/composer/vendor/bin/phpcs /usr/bin/
-
-    composer global require "phpunit/phpunit=8.*"
-    sudo ln -s ~/.config/composer/vendor/bin/phpunit /usr/bin/
-
-
-#### CentOS 7 or 8
-
-    sudo dnf install -y php-dom php-mbstring
-    pip3 install --user behave nose
-
-    composer global require "squizlabs/php_codesniffer=*"
-    sudo ln -s ~/.config/composer/vendor/bin/phpcs /usr/bin/
-
-    composer global require "phpunit/phpunit=^7"
-    sudo ln -s ~/.config/composer/vendor/bin/phpunit /usr/bin/
-
-## Run tests, code linter, code coverage
-
-See `README.md` in `test` subdirectory.

docs/mkdocs.yml

@@ -1,4 +1,4 @@
-site_name: Nominatim 3.5.2
+site_name: Nominatim Documentation
 theme: readthedocs
 docs_dir: ${CMAKE_CURRENT_BINARY_DIR}
 site_url: https://nominatim.org
@@ -16,8 +16,7 @@ pages:
         - 'FAQ': 'api/Faq.md'
     - 'Administration Guide':
         - 'Basic Installation': 'admin/Installation.md'
-        - 'Importing' : 'admin/Import.md'
-        - 'Updating' : 'admin/Update.md'
+        - 'Importing and Updating' : 'admin/Import-and-Update.md'
         - 'Advanced Installations' : 'admin/Advanced-Installations.md'
         - 'Migration from older Versions' : 'admin/Migration.md'
         - 'Troubleshooting' : 'admin/Faq.md'
@@ -26,7 +25,6 @@ pages:
         - 'OSM Data Import' : 'develop/Import.md'
         - 'Place Ranking' : 'develop/Ranking.md'
         - 'Postcodes' : 'develop/Postcodes.md'
-        - 'Setup Test Environment' : 'develop/Setup.md'
         - 'Documentation' : 'develop/Documentation.md'
     - 'External Data Sources':
         - 'Overview' : 'data-sources/overview.md'

lib/init.php

@@ -2,8 +2,3 @@
 
 require_once(CONST_BasePath.'/lib/lib.php');
 require_once(CONST_BasePath.'/lib/DB.php');
-
-if (get_magic_quotes_gpc()) {
-    echo "Please disable magic quotes in your php.ini configuration\n";
-    exit;
-}

nominatim/indexer/progress.py

@@ -0,0 +1,52 @@
+# SPDX-License-Identifier: GPL-2.0-only
+#
+# This file is part of Nominatim.
+# Copyright (C) 2020 Sarah Hoffmann
+
+import logging
+from datetime import datetime
+
+log = logging.getLogger()
+
+class ProgressLogger(object):
+    """ Tracks and prints progress for the indexing process.
+        `name` is the name of the indexing step being tracked.
+        `total` sets up the total number of items that need processing.
+        `log_interval` denotes the interval in seconds at which progres
+        should be reported.
+    """
+
+    def __init__(self, name, total, log_interval=1):
+        self.name = name
+        self.total_places = total
+        self.done_places = 0
+        self.rank_start_time = datetime.now()
+        self.next_info = 100 if log.isEnabledFor(logging.INFO) else total + 1
+
+    def add(self, num=1):
+        """ Mark `num` places as processed. Print a log message if the
+            logging is at least info and the log interval has past.
+        """
+        self.done_places += num
+
+        if self.done_places >= self.next_info:
+            now = datetime.now()
+            done_time = (now - self.rank_start_time).total_seconds()
+            places_per_sec = self.done_places / done_time
+            eta = (self.total_places - self.done_places)/places_per_sec
+
+            log.info("Done {} in {} @ {:.3f} per second - {} ETA (seconds): {:.2f}"
+                     .format(self.done_places, int(done_time),
+                             places_per_sec, self.name, eta))
+
+            self.next_info += int(places_per_sec)
+
+    def done(self):
+        """ Print final staticstics about the progress.
+        """
+        rank_end_time = datetime.now()
+        diff_seconds = (rank_end_time-self.rank_start_time).total_seconds()
+
+        log.warning("Done {}/{} in {} @ {:.3f} per second - FINISHED {}\n".format(
+                    self.done_places, self.total_places, int(diff_seconds),
+                    self.done_places/diff_seconds, self.name))

nominatim/nominatim.py

@@ -32,6 +32,8 @@ import psycopg2
 from psycopg2.extras import wait_select
 import select
 
+from indexer.progress import ProgressLogger
+
 log = logging.getLogger()
 
 def make_connection(options, asynchronous=False):
@@ -55,24 +57,19 @@ class RankRunner(object):
     def name(self):
         return "rank {}".format(self.rank)
 
-    def sql_index_sectors(self):
-        return """SELECT geometry_sector, count(*) FROM placex
+    def sql_count_objects(self):
+        return """SELECT count(*) FROM placex
                   WHERE rank_search = {} and indexed_status > 0
-                  GROUP BY geometry_sector
-                  ORDER BY geometry_sector""".format(self.rank)
+               """.format(self.rank)
 
-    def sql_nosector_places(self):
+    def sql_get_objects(self):
         return """SELECT place_id FROM placex
                   WHERE indexed_status > 0 and rank_search = {}
                   ORDER BY geometry_sector""".format(self.rank)
 
-    def sql_sector_places(self):
-        return """SELECT place_id FROM placex
-                  WHERE indexed_status > 0 and rank_search = {}
-                        and geometry_sector = %s""".format(self.rank)
-
-    def sql_index_place(self):
-        return "UPDATE placex SET indexed_status = 0 WHERE place_id = %s"
+    def sql_index_place(self, ids):
+        return "UPDATE placex SET indexed_status = 0 WHERE place_id IN ({})"\
+               .format(','.join((str(i) for i in ids)))
 
 
 class InterpolationRunner(object):
@@ -83,25 +80,19 @@ class InterpolationRunner(object):
     def name(self):
         return "interpolation lines (location_property_osmline)"
 
-    def sql_index_sectors(self):
-        return """SELECT geometry_sector, count(*) FROM location_property_osmline
-                  WHERE indexed_status > 0
-                  GROUP BY geometry_sector
-                  ORDER BY geometry_sector"""
+    def sql_count_objects(self):
+        return """SELECT count(*) FROM location_property_osmline
+                  WHERE indexed_status > 0"""
 
-    def sql_nosector_places(self):
+    def sql_get_objects(self):
         return """SELECT place_id FROM location_property_osmline
                   WHERE indexed_status > 0
                   ORDER BY geometry_sector"""
 
-    def sql_sector_places(self):
-        return """SELECT place_id FROM location_property_osmline
-                  WHERE indexed_status > 0 and geometry_sector = %s
-                  ORDER BY geometry_sector"""
-
-    def sql_index_place(self):
+    def sql_index_place(self, ids):
         return """UPDATE location_property_osmline
-                  SET indexed_status = 0 WHERE place_id = %s"""
+                  SET indexed_status = 0 WHERE place_id IN ({})"""\
+               .format(','.join((str(i) for i in ids)))
 
 
 class DBConnection(object):
@@ -210,83 +201,48 @@ class Indexer(object):
             self.index(RankRunner(rank))
 
         if self.maxrank == 30:
-            self.index(InterpolationRunner())
+            self.index(InterpolationRunner(), 20)
 
-        self.index(RankRunner(self.maxrank))
+        self.index(RankRunner(self.maxrank), 20)
 
-    def index(self, obj):
+    def index(self, obj, batch=1):
         """ Index a single rank or table. `obj` describes the SQL to use
-            for indexing.
+            for indexing. `batch` describes the number of objects that
+            should be processed with a single SQL statement
         """
         log.warning("Starting {}".format(obj.name()))
 
-        cur = self.conn.cursor(name='main')
-        cur.execute(obj.sql_index_sectors())
+        cur = self.conn.cursor()
+        cur.execute(obj.sql_count_objects())
 
-        total_tuples = 0
-        for r in cur:
-            total_tuples += r[1]
-        log.debug("Total number of rows; {}".format(total_tuples))
+        total_tuples = cur.fetchone()[0]
+        log.debug("Total number of rows: {}".format(total_tuples))
 
-        cur.scroll(0, mode='absolute')
+        cur.close()
 
         next_thread = self.find_free_thread()
-        done_tuples = 0
-        rank_start_time = datetime.now()
+        progress = ProgressLogger(obj.name(), total_tuples)
 
-        sector_sql = obj.sql_sector_places()
-        index_sql = obj.sql_index_place()
-        min_grouped_tuples = total_tuples - len(self.threads) * 1000
+        cur = self.conn.cursor(name='places')
+        cur.execute(obj.sql_get_objects())
 
-        next_info = 100 if log.isEnabledFor(logging.INFO) else total_tuples + 1
-
-        for r in cur:
-            sector = r[0]
-
-            # Should we do the remaining ones together?
-            do_all = done_tuples > min_grouped_tuples
-
-            pcur = self.conn.cursor(name='places')
-
-            if do_all:
-                pcur.execute(obj.sql_nosector_places())
-            else:
-                pcur.execute(sector_sql, (sector, ))
-
-            for place in pcur:
-                place_id = place[0]
-                log.debug("Processing place {}".format(place_id))
-                thread = next(next_thread)
-
-                thread.perform(index_sql, (place_id,))
-                done_tuples += 1
-
-                if done_tuples >= next_info:
-                    now = datetime.now()
-                    done_time = (now - rank_start_time).total_seconds()
-                    tuples_per_sec = done_tuples / done_time
-                    log.info("Done {} in {} @ {:.3f} per second - {} ETA (seconds): {:.2f}"
-                           .format(done_tuples, int(done_time),
-                                   tuples_per_sec, obj.name(),
-                                   (total_tuples - done_tuples)/tuples_per_sec))
-                    next_info += int(tuples_per_sec)
-
-            pcur.close()
-
-            if do_all:
+        while True:
+            places = [p[0] for p in cur.fetchmany(batch)]
+            if len(places) == 0:
                 break
 
+            log.debug("Processing places: {}".format(places))
+            thread = next(next_thread)
+
+            thread.perform(obj.sql_index_place(places))
+            progress.add(len(places))
+
         cur.close()
 
         for t in self.threads:
             t.wait()
 
-        rank_end_time = datetime.now()
-        diff_seconds = (rank_end_time-rank_start_time).total_seconds()
-
-        log.warning("Done {}/{} in {} @ {:.3f} per second - FINISHED {}\n".format(
-                 done_tuples, total_tuples, int(diff_seconds),
-                 done_tuples/diff_seconds, obj.name()))
+        progress.done()
 
     def find_free_thread(self):
         """ Generator that returns the next connection that is free for

settings/import-address.style

@@ -5,6 +5,11 @@
         "no" : "skip"
     }
 },
+{   "keys" : ["wikipedia", "wikipedia:*", "wikidata"],
+    "values" : {
+        "" : "extra"
+    }
+},
 {
     "keys" : ["name:prefix", "name:suffix", "name:botanical", "*wikidata"],
     "values" : {

settings/import-admin.style

@@ -1,4 +1,9 @@
 [
+{   "keys" : ["wikipedia", "wikipedia:*", "wikidata"],
+    "values" : {
+        "" : "extra"
+    }
+},
 {
     "keys" : ["name:prefix", "name:suffix", "name:botanical", "*wikidata"],
     "values" : {

settings/import-street.style

@@ -1,4 +1,9 @@
 [
+{   "keys" : ["wikipedia", "wikipedia:*", "wikidata"],
+    "values" : {
+        "" : "extra"
+    }
+},
 {
     "keys" : ["name:prefix", "name:suffix", "name:botanical", "*wikidata"],
     "values" : {

sql/functions/address_lookup.sql

@@ -272,7 +272,7 @@ BEGIN
   END IF;
 
   IF searchhousenumber IS NOT NULL THEN
-    location := ROW(in_place_id, null, null, hstore('ref', searchhousenumber),
+    location := ROW(null, null, null, hstore('ref', searchhousenumber),
                     'place', 'house_number', null, null, true, true, 28, 0)::addressline;
     RETURN NEXT location;
   END IF;

sql/functions/normalization.sql

@@ -81,7 +81,8 @@ BEGIN
   lookup_word := upper(trim(postcode));
   lookup_token := ' ' || make_standard_name(lookup_word);
   SELECT min(word_id) FROM word
-    WHERE word_token = lookup_token and class='place' and type='postcode'
+    WHERE word_token = lookup_token and word = lookup_word
+          and class='place' and type='postcode'
     INTO return_word_id;
   IF return_word_id IS NULL THEN
     return_word_id := nextval('seq_word');

sql/functions/place_triggers.sql

@@ -162,14 +162,14 @@ BEGIN
       IF st_area(NEW.geometry) < 0.000000001 AND st_area(existinggeometry) < 1 THEN
 
         -- re-index points that have moved in / out of the polygon, could be done as a single query but postgres gets the index usage wrong
-        update placex set indexed_status = 2 where indexed_status = 0 and 
-            (st_covers(NEW.geometry, placex.geometry) OR ST_Intersects(NEW.geometry, placex.geometry))
-            AND NOT (st_covers(existinggeometry, placex.geometry) OR ST_Intersects(existinggeometry, placex.geometry))
+        update placex set indexed_status = 2 where indexed_status = 0
+            AND ST_Intersects(NEW.geometry, placex.geometry)
+            AND NOT ST_Intersects(existinggeometry, placex.geometry)
             AND rank_search > existingplacex.rank_search AND (rank_search < 28 or name is not null);
 
-        update placex set indexed_status = 2 where indexed_status = 0 and 
-            (st_covers(existinggeometry, placex.geometry) OR ST_Intersects(existinggeometry, placex.geometry))
-            AND NOT (st_covers(NEW.geometry, placex.geometry) OR ST_Intersects(NEW.geometry, placex.geometry))
+        update placex set indexed_status = 2 where indexed_status = 0
+            AND ST_Intersects(existinggeometry, placex.geometry)
+            AND NOT ST_Intersects(NEW.geometry, placex.geometry)
             AND rank_search > existingplacex.rank_search AND (rank_search < 28 or name is not null);
 
       END IF;

sql/functions/placex_triggers.sql

@@ -455,9 +455,9 @@ BEGIN
   --    RAISE WARNING 'placex poly insert: % % % %',NEW.osm_type,NEW.osm_id,NEW.class,NEW.type;
 
         -- work around bug in postgis, this may have been fixed in 2.0.0 (see http://trac.osgeo.org/postgis/ticket/547)
-        update placex set indexed_status = 2 where (st_covers(NEW.geometry, placex.geometry) OR ST_Intersects(NEW.geometry, placex.geometry)) 
+        update placex set indexed_status = 2 where ST_Intersects(NEW.geometry, placex.geometry)
          AND rank_search > NEW.rank_search and indexed_status = 0 and ST_geometrytype(placex.geometry) = 'ST_Point' and (rank_search < 28 or name is not null or (NEW.rank_search >= 16 and address ? 'place'));
-        update placex set indexed_status = 2 where (st_covers(NEW.geometry, placex.geometry) OR ST_Intersects(NEW.geometry, placex.geometry)) 
+        update placex set indexed_status = 2 where ST_Intersects(NEW.geometry, placex.geometry)
          AND rank_search > NEW.rank_search and indexed_status = 0 and ST_geometrytype(placex.geometry) != 'ST_Point' and (rank_search < 28 or name is not null or (NEW.rank_search >= 16 and address ? 'place'));
       END IF;
     ELSE

sql/functions/utils.sql

@@ -431,7 +431,7 @@ DECLARE
   geo RECORD;
 BEGIN
   -- 10000000000 is ~~ 1x1 degree
-  FOR geo IN select quad_split_geometry(geometry, 0.25, 20) as geom LOOP
+  FOR geo IN select quad_split_geometry(geometry, 0.01, 20) as geom LOOP
     RETURN NEXT geo.geom;
   END LOOP;
   RETURN;
@@ -476,9 +476,9 @@ BEGIN
   IF placegeom IS NOT NULL AND ST_IsValid(placegeom) THEN
     IF ST_GeometryType(placegeom) in ('ST_Polygon','ST_MultiPolygon') THEN
       FOR geom IN select split_geometry(placegeom) FROM placex WHERE place_id = placeid LOOP
-        update placex set indexed_status = 2 where (st_covers(geom, placex.geometry) OR ST_Intersects(geom, placex.geometry)) 
+        update placex set indexed_status = 2 where ST_Intersects(geom, placex.geometry)
         AND rank_search > rank and indexed_status = 0 and ST_geometrytype(placex.geometry) = 'ST_Point' and (rank_search < 28 or name is not null or (rank >= 16 and address ? 'place'));
-        update placex set indexed_status = 2 where (st_covers(geom, placex.geometry) OR ST_Intersects(geom, placex.geometry)) 
+        update placex set indexed_status = 2 where ST_Intersects(geom, placex.geometry)
         AND rank_search > rank and indexed_status = 0 and ST_geometrytype(placex.geometry) != 'ST_Point' and (rank_search < 28 or name is not null or (rank >= 16 and address ? 'place'));
       END LOOP;
     ELSE

test/bdd/db/import/postcodes.feature

@@ -137,3 +137,22 @@ Feature: Import of postcodes
         And word contains
            | word  | class | type |
            | 01982 | place | postcode |
+
+    Scenario: Different postcodes with the same normalization can both be found
+        Given the places
+           | osm | class | type  | addr+postcode | addr+housenumber | geometry |
+           | N34 | place | house | EH4 7EA       | 111              | country:gb |
+           | N35 | place | house | E4 7EA        | 111              | country:gb |
+        When importing
+        Then location_postcode contains exactly
+           | country | postcode | geometry |
+           | gb      | EH4 7EA  | country:gb |
+           | gb      | E4 7EA   | country:gb |
+        When searching for "EH4 7EA"
+        Then results contain
+           | type     | placename |
+           | postcode | EH4 7EA      |
+        When searching for "E4 7EA"
+        Then results contain
+           | type     | placename |
+           | postcode | E4 7EA      |

test/bdd/steps/osm_data.py

@@ -4,13 +4,13 @@ import random
 import os
 from nose.tools import * # for assert functions
 
-@given(u'the (\d+ )?grid')
+@given(u'the ([0-9.]+ )?grid')
 def define_node_grid(context, grid_step):
     """
     Define a grid of node positions.
     """
     if grid_step is not None:
-        grid_step = int(grd_step.strip())
+        grid_step = float(grid_step.strip())
     else:
         grid_step = 0.00001
 

utils/check_import_finished.php

@@ -133,8 +133,7 @@ foreach ($aExpectedIndices as $sExpectedIndex) {
         $print_fail();
         echo <<< END
         Hints:
-        * Run './utils/setup.php --create-search-indices --ignore-errors' to
-          create missing indices.
+        * Rerun the setup.php --create-search-indices step
 
 END;
         exit(1);

utils/update.php

@@ -55,6 +55,7 @@ date_default_timezone_set('Etc/UTC');
 
 $oDB = new Nominatim\DB();
 $oDB->connect();
+$fPostgresVersion = $oDB->getPostgresVersion();
 
 $aDSNInfo = Nominatim\DB::parseDSN(CONST_Database_DSN);
 if (!isset($aDSNInfo['port']) || !$aDSNInfo['port']) $aDSNInfo['port'] = 5432;
@@ -90,13 +91,21 @@ if (isset($aDSNInfo['password']) && $aDSNInfo['password']) {
 if (!is_null(CONST_Osm2pgsql_Flatnode_File) && CONST_Osm2pgsql_Flatnode_File) {
     $oOsm2pgsqlCmd->addParams('--flat-nodes', CONST_Osm2pgsql_Flatnode_File);
 }
+if ($fPostgresVersion >= 11.0) {
+    $oOsm2pgsqlCmd->addEnvPair(
+        'PGOPTIONS',
+        '-c jit=off -c max_parallel_workers_per_gather=0'
+    );
+}
 
 
 $oIndexCmd = (new \Nominatim\Shell(CONST_BasePath.'/nominatim/nominatim.py'))
              ->addParams('--database', $aDSNInfo['database'])
              ->addParams('--port', $aDSNInfo['port'])
              ->addParams('--threads', $aResult['index-instances']);
-
+if (!$aResult['quiet']) {
+    $oIndexCmd->addParams('--verbose');
+}
 if ($aResult['verbose']) {
     $oIndexCmd->addParams('--verbose');
 }

vagrant/Install-on-Centos-7.sh

@@ -35,7 +35,7 @@
 
     sudo yum install -y postgresql11-server postgresql11-contrib postgresql11-devel \
                         postgis25_11 postgis25_11-utils \
-                        wget cmake make gcc gcc-c++ libtool policycoreutils-python \
+                        wget git cmake make gcc gcc-c++ libtool policycoreutils-python \
                         devtoolset-7 llvm-toolset-7 \
                         php-pgsql php php-intl libpqxx-devel \
                         proj-epsg bzip2-devel proj-devel boost-devel \
@@ -48,6 +48,18 @@
 
     pip3 install --user psycopg2 pytidylib
 
+# If you want to run the test suite, you need to install the following
+# additional packages:
+
+#DOCS:    :::sh
+    sudo yum install -y php-dom php-mbstring
+    pip3 install --user behave nose
+
+    composer global require "squizlabs/php_codesniffer=*"
+    sudo ln -s ~/.config/composer/vendor/bin/phpcs /usr/bin/
+
+    composer global require "phpunit/phpunit=7.*"
+    sudo ln -s ~/.config/composer/vendor/bin/phpunit /usr/bin/
 
 #
 # System Configuration
@@ -147,16 +159,23 @@ sudo sed -i 's:#.*::' /etc/httpd/conf.d/nominatim.conf #DOCS:
 # Building and Configuration
 # --------------------------
 #
-# Get the source code for the release and unpack it
+# Get the source code from Github and change into the source directory
 #
 if [ "x$1" == "xyes" ]; then  #DOCS:    :::sh
     cd $USERHOME
-    wget https://nominatim.org/release/Nominatim-3.5.2.tar.bz2
-    tar xf Nominatim-3.5.2.tar.bz2
+    git clone --recursive git://github.com/openstreetmap/Nominatim.git
+    cd Nominatim
 else                               #DOCS:
     cd $USERHOME/Nominatim         #DOCS:
 fi                                 #DOCS:
 
+# When installing the latest source from github, you also need to
+# download the country grid:
+
+if [ ! -f data/country_osm_grid.sql.gz ]; then       #DOCS:    :::sh
+    wget --no-verbose -O data/country_osm_grid.sql.gz https://www.nominatim.org/data/country_grid.sql.gz
+fi                                 #DOCS:
+
 # The code must be built in a separate directory. Create this directory,
 # then configure and build Nominatim in there:
 
@@ -164,7 +183,7 @@ fi                                 #DOCS:
     cd $USERHOME
     mkdir build
     cd build
-    cmake $USERHOME/Nominatim-3.5.2
+    cmake $USERHOME/Nominatim
     make
 
 #
@@ -175,10 +194,10 @@ fi                                 #DOCS:
 # with a web server accessible from the Internet. At a minimum the
 # following SELinux labeling should be done for Nominatim:
 
-    sudo semanage fcontext -a -t httpd_sys_content_t "$USERHOME/Nominatim-3.5.2/(website|lib|settings)(/.*)?"
+    sudo semanage fcontext -a -t httpd_sys_content_t "$USERHOME/Nominatim/(website|lib|settings)(/.*)?"
     sudo semanage fcontext -a -t httpd_sys_content_t "$USERHOME/build/(website|lib|settings)(/.*)?"
     sudo semanage fcontext -a -t lib_t "$USERHOME/build/module/nominatim.so"
-    sudo restorecon -R -v $USERHOME/Nominatim-3.5.2
+    sudo restorecon -R -v $USERHOME/Nominatim
     sudo restorecon -R -v $USERHOME/build
 
 
@@ -195,4 +214,4 @@ EOF
 
 
 # Nominatim is now ready to use. Continue with
-# [importing a database from OSM data](../admin/Import.md).
+# [importing a database from OSM data](../admin/Import-and-Update.md).

vagrant/Install-on-Centos-8.sh

@@ -26,21 +26,33 @@
 # Now you can install all packages needed for Nominatim:
 
 #DOCS:    :::sh
-    sudo dnf --enablerepo=PowerTools install -y postgresql12-server \
-                        postgresql12-contrib postgresql12-devel postgis30_12 \
-                        wget cmake make gcc gcc-c++ libtool policycoreutils-python-utils \
+    sudo dnf --enablerepo=PowerTools install -y postgresql10-server \
+                        postgresql10-contrib postgresql10-devel postgis25_10 \
+                        wget git cmake make gcc gcc-c++ libtool policycoreutils-python-utils \
                         llvm-toolset ccache clang-tools-extra \
                         php-pgsql php php-intl php-json libpq-devel \
-                        bzip2-devel proj-devel boost-devel \
+                        proj52-epsg bzip2-devel proj-devel boost-devel \
                         python3-pip python3-setuptools python3-devel \
                         expat-devel zlib-devel
 
     # make sure pg_config gets found
-    echo 'PATH=/usr/pgsql-12/bin:$PATH' >> ~/.bash_profile
+    echo 'PATH=/usr/pgsql-10/bin:$PATH' >> ~/.bash_profile
     source ~/.bash_profile
 
     pip3 install --user psycopg2 pytidylib
 
+# If you want to run the test suite, you need to install the following
+# additional packages:
+
+#DOCS:    :::sh
+    sudo dnf install -y php-dom php-mbstring
+    pip3 install --user behave nose
+
+    composer global require "squizlabs/php_codesniffer=*"
+    sudo ln -s ~/.config/composer/vendor/bin/phpcs /usr/bin/
+
+    composer global require "phpunit/phpunit=^7"
+    sudo ln -s ~/.config/composer/vendor/bin/phpunit /usr/bin/
 
 #
 # System Configuration
@@ -82,8 +94,8 @@ sudo chown vagrant /srv/nominatim  #DOCS:
 # with initializing the database, then enable the server to start at boot:
 
 
-    sudo /usr/pgsql-12/bin/postgresql-12-setup initdb
-    sudo systemctl enable postgresql-12
+    sudo /usr/pgsql-10/bin/postgresql-10-setup initdb
+    sudo systemctl enable postgresql-10
 
 #
 # Next tune the postgresql configuration, which is located in 
@@ -93,7 +105,7 @@ sudo chown vagrant /srv/nominatim  #DOCS:
 #
 # Now start the postgresql service after updating this config file.
 
-    sudo systemctl restart postgresql-12
+    sudo systemctl restart postgresql-10
 
 #
 # Finally, we need to add two postgres users: one for the user that does
@@ -141,25 +153,31 @@ sudo sed -i 's:#.*::' /etc/httpd/conf.d/nominatim.conf #DOCS:
 # Building and Configuration
 # --------------------------
 #
-# Get the source code for the release and unpack it
+# Get the source code from Github and change into the source directory
 #
 if [ "x$1" == "xyes" ]; then  #DOCS:    :::sh
     cd $USERHOME
-    wget https://nominatim.org/release/Nominatim-3.5.2.tar.bz2
-    tar xf Nominatim-3.5.2.tar.bz2
+    git clone --recursive git://github.com/openstreetmap/Nominatim.git
+    cd Nominatim
 else                               #DOCS:
     cd $USERHOME/Nominatim         #DOCS:
 fi                                 #DOCS:
 
+# When installing the latest source from github, you also need to
+# download the country grid:
+
+if [ ! -f data/country_osm_grid.sql.gz ]; then       #DOCS:    :::sh
+    wget --no-verbose -O data/country_osm_grid.sql.gz https://www.nominatim.org/data/country_grid.sql.gz
+fi                                 #DOCS:
+
 # The code must be built in a separate directory. Create this directory,
-# then configure and build Nominatim in there. Note that you need to
-# tell cmake about the more recent Postgresql version that you use:
+# then configure and build Nominatim in there:
 
 #DOCS:    :::sh
     cd $USERHOME
     mkdir build
     cd build
-    cmake -DPostgreSQL_ADDITIONAL_VERSIONS=12 $USERHOME/Nominatim-3.5.2
+    cmake $USERHOME/Nominatim
     make
 
 #
@@ -170,10 +188,10 @@ fi                                 #DOCS:
 # with a web server accessible from the Internet. At a minimum the
 # following SELinux labeling should be done for Nominatim:
 
-    sudo semanage fcontext -a -t httpd_sys_content_t "$USERHOME/Nominatim-3.5.2/(website|lib|settings)(/.*)?"
+    sudo semanage fcontext -a -t httpd_sys_content_t "$USERHOME/Nominatim/(website|lib|settings)(/.*)?"
     sudo semanage fcontext -a -t httpd_sys_content_t "$USERHOME/build/(website|lib|settings)(/.*)?"
     sudo semanage fcontext -a -t lib_t "$USERHOME/build/module/nominatim.so"
-    sudo restorecon -R -v $USERHOME/Nominatim-3.5.2
+    sudo restorecon -R -v $USERHOME/Nominatim
     sudo restorecon -R -v $USERHOME/build
 
 
@@ -190,4 +208,4 @@ EOF
 
 
 # Nominatim is now ready to use. Continue with
-# [importing a database from OSM data](../admin/Import.md).
+# [importing a database from OSM data](../admin/Import-and-Update.md).

vagrant/Install-on-Ubuntu-18-nginx.sh

@@ -24,7 +24,7 @@ export DEBIAN_FRONTEND=noninteractive
                             postgresql-contrib-10 \
                             nginx php-fpm php php-pgsql \
                             php-intl python3-setuptools python3-dev python3-pip \
-                            python3-psycopg2 python3-tidylib
+                            python3-psycopg2 python3-tidylib git
 
     export USERNAME=vagrant
     export USERHOME=/home/vagrant

vagrant/Install-on-Ubuntu-18.sh

@@ -31,8 +31,20 @@ export DEBIAN_FRONTEND=noninteractive #DOCS:
                             postgresql-contrib-10 postgresql-10-postgis-scripts \
                             apache2 php php-pgsql libapache2-mod-php \
                             php-intl python3-setuptools python3-dev python3-pip \
-                            python3-psycopg2 python3-tidylib
+                            python3-psycopg2 python3-tidylib git
 
+# If you want to run the test suite, you need to install the following
+# additional packages including the PHP package manager composer:
+
+    pip3 install --user behave nose
+
+    sudo apt-get install -y composer php-cgi php-cli php-mbstring php-xml zip unzip
+
+    composer global require "squizlabs/php_codesniffer=*"
+    sudo ln -s ~/.config/composer/vendor/bin/phpcs /usr/bin/
+
+    composer global require "phpunit/phpunit=8.*"
+    sudo ln -s ~/.config/composer/vendor/bin/phpunit /usr/bin/
 
 #
 # System Configuration
@@ -96,14 +108,14 @@ export DEBIAN_FRONTEND=noninteractive #DOCS:
 
 #DOCS:```sh
 sudo tee /etc/apache2/conf-available/nominatim.conf << EOFAPACHECONF
-<Directory "$USERHOME/build/website">
+<Directory "$USERHOME/build/website"> #DOCS:<Directory "$USERHOME/Nominatim/build/website">
   Options FollowSymLinks MultiViews
   AddType text/html   .php
   DirectoryIndex search.php
   Require all granted
 </Directory>
 
-Alias /nominatim $USERHOME/build/website
+Alias /nominatim $USERHOME/build/website  #DOCS:Alias /nominatim $USERHOME/Nominatim/build/website
 EOFAPACHECONF
 #DOCS:```
 
@@ -123,23 +135,30 @@ sudo sed -i 's:#.*::' /etc/apache2/conf-available/nominatim.conf #DOCS:
 # Building and Configuration
 # --------------------------
 #
-# Get the source code for the release and unpack it
+# Get the source code from Github and change into the source directory
 #
 if [ "x$1" == "xyes" ]; then  #DOCS:    :::sh
     cd $USERHOME
-    wget https://nominatim.org/release/Nominatim-3.5.2.tar.bz2
-    tar xf Nominatim-3.5.2.tar.bz2
+    git clone --recursive git://github.com/openstreetmap/Nominatim.git
+    cd Nominatim
 else                               #DOCS:
     cd $USERHOME/Nominatim         #DOCS:
 fi                                 #DOCS:
 
+# When installing the latest source from github, you also need to
+# download the country grid:
+
+if [ ! -f data/country_osm_grid.sql.gz ]; then       #DOCS:    :::sh
+    wget -O data/country_osm_grid.sql.gz https://www.nominatim.org/data/country_grid.sql.gz
+fi                                 #DOCS:
+
 # The code must be built in a separate directory. Create this directory,
 # then configure and build Nominatim in there:
 
     cd $USERHOME                   #DOCS:    :::sh
     mkdir build
     cd build
-    cmake $USERHOME/Nominatim-3.5.2
+    cmake $USERHOME/Nominatim
     make
 
 # You need to create a minimal configuration file that tells nominatim
@@ -154,4 +173,4 @@ EOF
 
 
 # Nominatim is now ready to use. Continue with
-# [importing a database from OSM data](../admin/Import.md).
+# [importing a database from OSM data](../admin/Import-and-Update.md).

vagrant/Install-on-Ubuntu-20.sh

@@ -6,6 +6,12 @@ sudo update-locale LANG=en_US.UTF-8 #DOCS:
 export APT_LISTCHANGES_FRONTEND=none #DOCS:
 export DEBIAN_FRONTEND=noninteractive #DOCS:
 
+#
+# !!! danger "Important"
+#     Ubuntu 20.04 uses Postgresql 12 and Postgis 3, which are known to cause
+#     performance issues. They are not recommended for a production
+#     installation at the moment.
+#
 # *Note:* these installation instructions are also available in executable
 #         form for use with vagrant under vagrant/Install-on-Ubuntu-20.sh.
 #
@@ -33,8 +39,14 @@ export DEBIAN_FRONTEND=noninteractive #DOCS:
                             postgresql-contrib postgresql-12-postgis-3-scripts \
                             apache2 php php-pgsql libapache2-mod-php \
                             php-intl python3-setuptools python3-dev python3-pip \
-                            python3-psycopg2 python3-tidylib
+                            python3-psycopg2 python3-tidylib git
 
+# If you want to run the test suite, you need to install the following
+# additional packages:
+
+    sudo apt-get install -y phpunit php-codesniffer php-cgi
+
+    pip3 install --user behave nose
 #
 # System Configuration
 # ====================
@@ -97,14 +109,14 @@ export DEBIAN_FRONTEND=noninteractive #DOCS:
 
 #DOCS:```sh
 sudo tee /etc/apache2/conf-available/nominatim.conf << EOFAPACHECONF
-<Directory "$USERHOME/build/website">
+<Directory "$USERHOME/build/website"> #DOCS:<Directory "$USERHOME/Nominatim/build/website">
   Options FollowSymLinks MultiViews
   AddType text/html   .php
   DirectoryIndex search.php
   Require all granted
 </Directory>
 
-Alias /nominatim $USERHOME/build/website
+Alias /nominatim $USERHOME/build/website  #DOCS:Alias /nominatim $USERHOME/Nominatim/build/website
 EOFAPACHECONF
 #DOCS:```
 
@@ -124,23 +136,30 @@ sudo sed -i 's:#.*::' /etc/apache2/conf-available/nominatim.conf #DOCS:
 # Building and Configuration
 # --------------------------
 #
-# Get the source code for the release and unpack it
+# Get the source code from Github and change into the source directory
 #
 if [ "x$1" == "xyes" ]; then  #DOCS:    :::sh
     cd $USERHOME
-    wget https://nominatim.org/release/Nominatim-3.5.2.tar.bz2
-    tar xf Nominatim-3.5.2.tar.bz2
+    git clone --recursive git://github.com/openstreetmap/Nominatim.git
+    cd Nominatim
 else                               #DOCS:
     cd $USERHOME/Nominatim         #DOCS:
 fi                                 #DOCS:
 
+# When installing the latest source from github, you also need to
+# download the country grid:
+
+if [ ! -f data/country_osm_grid.sql.gz ]; then       #DOCS:    :::sh
+    wget -O data/country_osm_grid.sql.gz https://www.nominatim.org/data/country_grid.sql.gz
+fi                                 #DOCS:
+
 # The code must be built in a separate directory. Create this directory,
 # then configure and build Nominatim in there:
 
-    cd $USERHOME
+    cd $USERHOME                   #DOCS:    :::sh
     mkdir build
     cd build
-    cmake $USERHOME/Nominatim-3.5.2
+    cmake $USERHOME/Nominatim
     make
 
 # You need to create a minimal configuration file that tells nominatim
@@ -155,4 +174,4 @@ EOF
 
 
 # Nominatim is now ready to use. Continue with
-# [importing a database from OSM data](../admin/Import.md).
+# [importing a database from OSM data](../admin/Import-and-Update.md).